PaperlayerView exact revision on arXiv

Source-linked AI summary

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu

arXiv:2410.10813v2cs.CL

TL;DR

Long-term memory is underexplored despite its importance for personalized chat assistants and sustained user-AI interactions. The paper introduces LONGMEMEVAL and a unified indexing, retrieval, and reading framework, finding substantial difficulty in current systems and improvements from targeted memory designs.

  • Problem

    Existing evaluations provide limited evidence about chat assistants’ long-term memory across sustained, task-oriented interactions and dynamic memory abilities.

  • Method

    The paper introduces 500-question LONGMEMEVAL and analyzes memory designs across indexing, retrieval, and reading stages.

  • Results

    Current systems exhibit substantial performance drops on LONGMEMEVAL, while session decomposition, fact-augmented key expansion, and time-aware query expansion improve memory recall and question answering.

  • Takeaways & Limitations

    LONGMEMEVAL provides a benchmark and design guidance for advancing long-term memory in LLM-based chat assistants.

  • Takeaways & Limitations

    The retrieval-stage rank-merging strategy underperformed indexing-stage merging in the reported experiments.

Abstract

from arXiv · show

Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval. Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https://github.com/xiaowu0162/LongMemEval.

1 INTRODUCTION

Long-term memory is essential for personalized chat assistants, but existing benchmarks inadequately capture sustained user-assistant interactions and dynamic memory abilities. LONGMEMEVAL addresses these gaps and identifies effective designs across indexing, retrieval, and reading.

  • Motivation: Long-term memory supports assistants in recalling and reasoning over user information accumulated across interactions, improving personalization and response accuracy.Failing to incorporate user background and preferences can diminish response accuracy and user satisfaction.
  • Benchmark gap: Existing benchmarks underrepresent task-oriented user-assistant interactions, use limited history lengths, and insufficiently test cross-session and temporal reasoning.These shortcomings limit evaluation difficulty and coverage of dynamic long-term memory abilities.
  • LONGMEMEVAL: 500 manually created questions evaluate information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention in scalable task-oriented chat histories.Two standard settings contain approximately 115k tokens per problem or 500 sessions totaling around 1.5 million tokens.
  • LONGMEMEVAL: LONGMEMEVAL exposes substantial difficulty: long-context LLMs show a 30%∼60% performance drop on the approximately 115k-token setting.The benchmark requires online memorization during interactions and question answering after all sessions.
  • Memory design: Round-level storage is more effective overall than session-level storage, while individual-fact compression harms overall performance but improves multi-session reasoning accuracy.The findings identify value granularity as an important memory design choice.
  • Memory design: Fact-augmented key expansion raises memory recall by 9.4% recall@k and downstream question-answering accuracy by 5.4%.Time-aware indexing and query expansion improve temporal-reasoning recall by 6.8%∼11.3% when a strong LLM performs query expansion.
  • Memory design: Accurately using retrieved items remains non-trivial even with perfect memory recall, while Chain-of-Note and structured data formats improve question-answering accuracy by as much as 10 absolute points.This result separates retrieval quality from the reading stage of memory-augmented answering.

2 RELATED WORK

Prior long-term dialogue benchmarks increasingly use question answering, but they often omit capabilities and interaction settings central to sustained user-assistant memory. LONGMEMEVAL broadens coverage across question types, history depth, and dialogue structure.

  • Long-term dialogue benchmarks: Long-term dialogue evaluation progressed from personalized response generation toward question answering to measure memory accuracy more directly.The related benchmarks include both human-human and human-AI chat histories.
  • Benchmark comparison: Table 1 compares benchmarks by dialogue type, session and question counts, history context depth, and coverage of information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention.The table defines #Sess, #Q, and context depth and notes approximated or bounded values where applicable.
  • Long-term dialogue benchmarks: Existing QA benchmarks overlook synthesizing information across many sessions, recalling assistant side information, updating user details, and reasoning about complex temporal references.They also often use brief histories that do not reflect task-oriented interactions.
  • Long-term memory methods: Long-term memory methods include processing extensive histories as long-context inputs and compressing histories into internal representations, discrete tokens, or retrievable text segments.LONGMEMEVAL can evaluate any memory system, while the paper studies online context compression.

3 LONGMEMEVAL

LONGMEMEVAL evaluates long-term memory in realistic, scalable user-assistant histories across five core abilities. Its curated construction and pilot studies expose substantial weaknesses in current commercial systems and long-context LLMs.

  • Problem formulation: The benchmark represents each evaluation instance as timestamped chat sessions, a user question with a later date, and a short answer or open-ended rubric.Histories are presented session by session during testing.
  • Benchmark scope: LONGMEMEVAL contains 500 manually created questions covering information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.Each question requires recalling information hidden in one or more task-oriented dialogues.
  • Data construction: Human experts curate questions and evidence statements, while LLM-simulated evidence sessions are human-edited before unrelated sessions are combined into a configurable-length history.Evidence may be conveyed indirectly, and its position is annotated for quality control.
  • Evaluation: LONGMEMEVAL assesses answers with an LLM evaluator rather than exact matching, achieving more than 97% agreement with human experts.It also supports Recall@k and NDCG@k when systems expose retrieval results.
  • Pilot findings: 30% to 60% performance decline occurred when four long-context LLMs read approximately 115k-token histories instead of oracle-retrieved evidence sessions.The decline persisted regardless of whether chain-of-note was applied.

4 A UNIFIED VIEW OF LONG-TERM MEMORY ASSISTANTS

The paper models long-term memory assistants as a three-stage key-value system spanning indexing, retrieval, and reading, with four controllable design points. Experiments motivate finer-grained values, fact-augmented keys, time-aware retrieval, and improved reading strategies.

  • Framework: The unified framework organizes long-term memory into indexing, retrieval, and reading stages, with value, key, query, and reading-strategy control points.It presents a common view of existing memory-augmented assistants and analyzes design choices across these stages.
  • Value: Round-level storage is generally more effective than session-level storage, while compressing into individual facts can lose information despite helping multi-session reasoning.The value control point concerns the format and granularity of stored history.
  • Reading strategy: Extracting key information before answering and using structured-format prompting are proposed reading optimizations because retrieval alone does not ensure reasoning over extensive context.The framework treats reading as a separate control point from retrieval.

5 EXPERIMENT RESULTS

Experiments show that memory representation, key construction, temporal query handling, and reading format all materially affect retrieval and question answering. The strongest gains come from fact-augmented indexing and time-aware query expansion, while reader capability determines useful context length.

  • Value: Round-level storage is generally more effective than session-level storage, while fact-level compression loses information but improves multi-session reasoning.Decomposing sessions into rounds significantly enhances reading with GPT-4o, whereas summaries or facts generally hurt QA performance.
  • Reading: GPT-4o continues improving beyond 20k retrieved tokens, whereas Llama 3.1 8B Instruct declines sharply beyond 3k tokens.The optimal retrieved-token budget therefore varies with reader capability.
  • Indexing: Condensed summaries, keyphrases, or user facts alone do not improve memory recall, likely because the retriever already handles long-text semantics effectively.Document expansion preserves both compressed highlights and the completeness of the original value.
  • Indexing: 9.4% higher recall@k and 5.4% higher accuracy result from expanding value keys with extracted user facts.The expansion concatenates compressed user facts with the original value during indexing.
  • Query: 11.3% higher recall with rounds and 6.8% higher recall with sessions follow time-aware query expansion on temporal reasoning.The method indexes event dates and filters retrieval using an LLM-inferred query time range.
  • Reading: A suboptimal reading strategy causes up to a 10-point absolute performance drop for GPT-4o even under oracle retrieval.With Chain-of-Note, JSON consistently benefits readers of different capabilities; without it, JSON is not consistently better than natural language.

6 CONCLUSION

The paper presents LongMemEval as a challenging benchmark for five long-term memory abilities and reports design strategies that improve memory recall and question answering. It argues that more sophisticated memory mechanisms are needed for personalized and reliable conversational AI.

  • LongMemEval evaluates information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention in chat assistants.The benchmark is described as comprehensive and challenging.
  • The proposed session decomposition, fact-augmented key expansion, and time-aware query expansion improve memory recall and question answering performance.These strategies arise from analyzing indexing, retrieval, and reading design choices.
  • The findings identify a need for more sophisticated memory mechanisms to support personalized and reliable conversational AI.

REPRODUCIBILITY STATEMENT

The authors document benchmark construction and plan to release fixed evaluation datasets and algorithms to support replication and reuse.

  • The benchmark construction process, prompting attributes, and instructions are documented in the paper and Appendix A.1.
  • Two fixed evaluation datasets, LONGMEMEVALS and LONGMEMEVALM, and the algorithm are planned for public release.

ETHICS STATEMENT

The ethics process addresses dataset licensing, privacy and offensiveness screening, annotator expertise, review procedures, labor, and compensation.

  • The released dataset uses source data with Apache 2.0 and MIT licenses, while the planned release license is MIT.
  • Three expert NLP annotators screened newly created questions and evidence sessions for personally identifiable information and offensive content.They also resolved uncertain cases through discussion.
  • Approximately 400 human hours supported dataset construction and 150 hours supported the commercial-systems study, with annotators paid for annotation work.

A.1 DATASET CONSTRUCTION

LONGMEMEVAL constructs scalable, task-oriented chat histories around manually curated questions and user backgrounds. Its pipeline samples diverse sessions, resolves timestamps, and places evidence in varied locations to test long-term memory under realistic difficulty.

  • Attribute Ontology: LONGMEMEVAL uses a manually constructed ontology spanning demographic information, lifestyle, situational context, life events, and belongings.
  • Background Sampling: Llama 3 70B Instruct generates detailed user backgrounds from sampled attributes, which support subsequent question construction.
  • Question Construction: Questions are generated for each memory ability and then manually filtered and edited after model-based proposal.
  • Evidence Session Construction: Evidence sessions are simulated as concise, indirect user-assistant dialogues, with expert annotators editing roughly 70% to enforce evidence and style constraints.
  • History Construction: A three-stage pipeline constructs coherent, freely extensible histories through session-pool construction, session sampling, and timestamp resolution.
  • Dataset Statistics: Most questions require evidence from multiple sessions, with evidence statements distributed diversely within sessions, increasing the benchmark’s challenge.

B A HUMAN STUDY ON COMMERCIAL MEMORY CHATBOTS

The human study evaluates ChatGPT and Coze on manually assembled multi-session histories. Both systems handle some single-session memorization but show substantial declines when answers require aggregation across sessions.

  • Study Setup: The study evaluates ChatGPT and Coze on 97 questions using manually administered, short histories of 3–6 sessions.
  • Study Setup: Annotators interacted with each chatbot session by session, posed the question in a fresh session, judged correctness, and cleared memory between instances.
  • Results: Both systems answer a considerable number of single-session information-extraction problems correctly but exhibit significant drops on other question types requiring multi-session aggregation.
  • Unified Framework: The unified framework represents long-term memory as a key-value datastore processed through indexing, querying, selection, and reading functions.
  • System Analysis: Most surveyed systems use flat retrieval and direct reading, while the paper identifies extract-before-read as important for high performance.

D MEMORY OPTIMIZATIONS: IMPLEMENTATION DETAILS

The paper implements memory optimizations across value decomposition, key expansion, time-aware retrieval, and reading. Additional evaluations test these designs across models and settings.

  • Value Decomposition: Value decomposition compresses sessions or rounds into summaries, keyphrases, or user facts using LLM-based extraction.
  • Key Expansion: Key expansion prepends extracted summaries, keyphrases, or user facts to the corresponding indexing key.
  • Time-Aware Indexing and Query Expansion: Time-aware indexing extracts timestamped events, while query expansion extracts a time range for questions focused on a specific period.
  • Reading Strategy: The reading prompt asks the model to traverse recalled documents, extract evidence, and then generate an answer, following the Chain-of-Note idea.
  • Additional Analyses: Additional experiments evaluate more LLMs, retrievers, memory optimizations, and end-to-end retrieval-augmented generation errors.
  • Additional Analyses: Across five additional LLMs, long-context reading remains below oracle retrieval, while fact-based key expansion and Chain-of-Note consistently improve performance.

E.2 ABLATIONS ON RETRIEVER SELECTION

Retriever ablations compare dense and sparse retrieval under vanilla and expanded-key designs, while further analyses examine rank merging, temporal query extraction, and reader errors.

  • Retriever Comparison: Both dense retrieval embeddings significantly outperform BM25 sparse retrieval across most settings.
  • Key Expansion: Fact-based key expansion consistently improves performance over directly using the value as the indexing key.
  • Key Expansion: Summary and keyphrase expansion help in some settings, but fact expansion produces the greatest performance gain.
  • Rank Merging: Rank merging performs much worse than key merging, while key merging highlights extracted information without multiplying index size by m+1.
  • Error Analysis: Correct retrieval is necessary for approximately 90% of correct answers, leaving reading strategy as a substantial remaining source of error.
Loading 2410.10813v2…