PaperlayerView exact revision on arXiv

Source-linked AI summary

MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants

Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, Ji-Rong Wen

arXiv:2409.20163v1cs.AIcs.CL

TL;DR

Objective, automatic evaluation of personal-assistant memory lacks scalable, reliable QAs from user messages, while LLM-generated ground truths can be inaccurate. MemSim uses BRNet and causal generation to create MemDaily, and the paper reports QA accuracy and benchmarks different memory mechanisms.

  • Problem

    Objective, automatic evaluation lacks scalable, reliable QAs from user messages, while LLM-generated ground truths can be inaccurate and simulated profiles can lack diversity.

    Source: §1 · pp. 1–2

  • Method

    MemSim combines BRNet-generated hierarchical user profiles with causal generation to construct user messages and QAs for memory evaluation.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    MemSim significantly ensures accuracy of constructed QAs; FullMem and RetrMem outperform other memory mechanisms across both datasets.

    Source: §4.3 · p. 8, §5.2 · p. 9

  • Takeaways & Limitations

    MemDaily provides a daily-life dataset and benchmark for evaluating memory mechanisms in LLM-based agents.

    Source: §1 · p. 2

    BRNet assumes a loop-free causal structure, forming a directed acyclic graph.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim.

1 Introduction

Evaluating personal-assistant memory requires reliable recall of facts from prior messages, but existing approaches face limits in scalability, QA reliability, and profile diversity. MemSim addresses these challenges with a Bayesian simulator and causal generation, and the paper introduces MemDaily as a daily-life dataset and benchmark for memory mechanisms.

1 Introduction

  • Personal assistants must store factual information from earlier messages and recall relevant details to answer later personal queries.The introduction illustrates this with recalling a movie seat from a prior message.Source: §1 · p. 1
  • Objective, automatic evaluation of memory remains limited; manual QA annotation requires substantial labor, while LLM-generated QAs risk factual hallucinations and lack profile diversity.The paper reports that vanilla-LLM ground-truth correctness is below 90% in most scenarios and below 40% in some complex scenarios.Source: §1 · pp. 1–2
  • MemSim is a Bayesian simulator designed to construct reliable, diverse, and scalable QAs from generated user messages for evaluating personal-assistant memory.The paper presents BRNet for simulated users and a causal generation mechanism for user messages and QAs.Source: §1 · p. 2
  • The paper analyzes dataset-construction challenges across reliability, diversity, and scalability, then addresses them with BRNet and causal generation.BRNet generates simulated users; the causal mechanism constructs their messages and QAs.Source: §1 · p. 2
  • The authors create MemDaily as a daily-life dataset, assess its quality, and provide a benchmark for comparing memory mechanisms in LLM-based agents.The project is released publicly.Source: §1 · p. 2

2 Related Works

Prior work develops memory mechanisms for personal assistants and evaluates memory through human ratings or constructed QA pairs. The paper identifies remaining needs for automatic QA construction and distinguishes personal-message memory evaluation from knowledge-based QA datasets.

2 Related Works

  • Prior personal-assistant memory methods store and organize user information, including essential facts in memory pools and daily events in hierarchical summaries.The cited examples are MPC [9] and MemoryBank [11].Source: §2 · p. 2
  • Memory evaluations use subjective human scoring or objective dialogue QA pairs, but both approaches can still require human involvement.Human scoring can be costly and susceptible to annotator variation [9] [11] [17]; QA construction or editing can require human input [13] [18] [19].Source: §2 · p. 2
  • Knowledge-based QA datasets typically target common-sense questions, omit user messages and retrieval indexes, and can depend on extracted entities, limiting scalability.The paper contrasts these datasets with personal questions whose answers depend on messages in the same trajectory [23] [24] [25] [29].Source: §2 · pp. 2–3

3 Methods

MemSim builds an automated memory-evaluation pipeline from sampled hierarchical user profiles, using a Bayesian network and shared-hint causal generation to construct messages and QAs. The paper applies it to MemDaily, a daily-life dataset spanning six QA types.

3.1 Overview of MemSim

  • MemSim models user entities and attributes with BRNet, then uses a causal mechanism to generate evaluation messages and QAs from sampled profiles.The pipeline is intended to support objective, automatic evaluation of personal assistants’ memory.Source: §3 · p. 3, §3.1 · p. 3

3.2 Bayesian Relation Network

  • BRNet models dependencies among user entities’ attributes as a loop-free DAG, then uses ancestral sampling to produce hierarchical profiles without computing the joint distribution.The graph can incorporate scenario-specific prior knowledge, and its vertices, edges, and conditional distributions can be scaled across scenarios.Source: §3.2 · pp. 3–5

3.3 Causal Generation Mechanism

  • Shared profile-derived hints drive message rewriting and five QA forms—single-hop, multi-hop, comparative, aggregative, and post-processing—keeping factual content aligned.The method also adds entity-side and attribute-side noise to messages; each trajectory serves as an evaluation test instance.Source: §3.3 · pp. 5–6

3.4 MemDaily: A Dataset in the Daily-life Scenario

  • MemDaily covers daily-life scenarios with 11 entities, 73 attributes, and six subdatasets spanning distinct QA types.The six subsets are Simple, Conditional, Comparative, Aggregative, Post-processing, and Noisy.Source: §3.4 · p. 6

4 Evaluations

MemSim’s evaluations find that generated profiles and messages are diverse, while constructed QAs have high ground-truth accuracy. Human verification also identifies occasional accuracy loss from LLM rewriting.

4.1 Evaluation on User Profiles

  • MemSim outperforms profile-generation baselines on R-Human and achieves the highest profile diversity, although R-Human and R-GPT scores are inconsistent.The authors suggest the inconsistency may stem from inaccurate LLM scoring.Source: §4.1 · p. 7

4.2 Evaluation on User Messages

  • Despite rigorous QA-construction constraints, MemSim maintains relatively high user-message scores and has the highest diversity index.The authors attribute the diversity result to BRNet and the causal generation mechanism.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Evaluation on Questions and Answers

  • Human evaluators verify ground truths in approximately 20% of MemDaily trajectories across textual answers, single-choice answers, and retrieval targets; accuracy is high, with occasional deviations from LLM rewriting.The evaluation also reports lower reliability for OracleMem, which directly generates answers from targeted messages and questions.Source: §4.3 · p. 8, Table 5 · p. 8

5 Benchmark

The benchmark varies the amount of irrelevant daily-life material and evaluates memory mechanisms on factual QA, target retrieval, and time costs. FullMem and RetrMem achieve high factual-QA accuracy, while retrieval and efficiency patterns depend on context length and method.

5.1 Experimental Settings

  • MemDaily-vanilla has no added irrelevant posts, while MemDaily-η adds them at controlled proportions; larger η means greater difficulty, with vanilla and MemDaily-100 emphasized.The benchmark also evaluates MemDaily-10, MemDaily-50, and MemDaily-200 in the appendix.Source: §5.1 · p. 9
  • The benchmark measures effectiveness with answer accuracy and Recall@5, and efficiency with response time and adaptation time.It compares full, recent, retrieved, no, noisy, and oracle memory configurations.Source: §5.1 · p. 9

5.2 Effectiveness of Memory Mechanisms

  • FullMem and RetrMem achieve high factual-question accuracy across both datasets and outperform the other memory mechanisms.ReceMem tends to underperform with many noisy messages; agents also struggle more with comparative and aggregative questions.Source: §5.2 · p. 9, Table 6 · p. 10
  • LLM retrieval performs best in short contexts, whereas Embedding achieves higher Recall@5 in longer contexts.Separating retrieval and inference may perform differently from integrating them.Source: Table 7 · p. 10, §5.2 · p. 10

5.3 Efficiency of Memory Mechanisms

  • RetrMem takes the most response time in short contexts, FullMem’s response time rises faster as context lengthens, and RetrMem needs substantially more adaptation time.The authors attribute RetrMem’s adaptation cost to building FAISS indexes.Source: §5.3 · p. 10

6 Limitations and Conclusions

The paper introduces MemSim and uses it to create MemDaily and a benchmark of memory mechanisms. Its evaluation is limited to factual information and non-dialogue settings.

6 Limitations and Conclusions

  • MemSim combines BRNet and causal generation to create MemDaily and support a benchmark evaluating LLM-agent memory mechanisms.The paper presents the work as an initial study.Source: §6 · p. 10
  • The evaluation covers factual information, not higher-level abstract information such as hidden hobbies or preferences, and excludes dialogue forms.The authors identify both as issues for future work.Source: §6 · p. 10

A Proof in Bayesian Relation Network

The appendix proves two properties of BRNet: its joint distribution factorizes, and ancestral sampling yields the same result as sampling from that joint distribution.

A.1 Proof of Theorem 1

  • Theorem 1 derives BRNet’s joint-distribution factorization by reversing a topological ordering and applying conditional probability with Assumption 1.The proof uses the DAG structure and the stated condition on descendant variables.Source: §A.1 · p. 14

A.2 Proof of Theorem 2

  • Theorem 2 establishes that ancestral sampling from BRNet is equivalent to sampling from its joint probability distribution.The proof calculates a reversed topological ordering and derives the equivalence using Assumption 2.Source: §A.2 · p. 14

B Extensive Evaluation on User Messages by GPT-4o

This section uses GPT-4o to score user messages as a reference and presents the results in Table 10.

B Extensive Evaluation on User Messages by GPT-4o

C.1 Results on MemDaily-10

The section reports MemDaily-10 results for accuracy, recall@5, response time, and adaptation time.

C.1 Results on MemDaily-10

C.2 Results of MemDaily-50

The section reports MemDaily-50 results for accuracy, recall@5, response time, and adaptation time.

C.2 Results of MemDaily-50

C.3 Results of MemDaily-200

The section reports four evaluation measures for MemDaily-200 and presents case studies of generated profiles, user messages, and question-answer examples.

C.3 Results of MemDaily-200

D Case Studies

  • The case studies present hierarchical profiles generated from BRNet, generated user-message examples, and question-answer examples for each type.Source: D Case Studies · p. 19

D.1 Case Study on Generated User Profiles

  • MemDaily includes 11 entities across 7 types and 73 attributes, summarized in Table 23.Source: §D.1 · p. 19, Table 23 · p. 21
  • Scenario-based attribute constraints are encoded as causal relations in BRNet; observations of 50 generated profiles found most aligned with real-world users without contradictions.Source: §D.1 · pp. 19–20

D.2 Case Study on User Messages

The case studies show generated user messages and their corresponding hints, then illustrate several QA types and retrieval targets. The authors report that their mechanisms ensure no contradictions among user messages.

D.2 Case Study on User Messages

  • The user-message case study pairs generated messages with hints and reports that the mechanisms ensure no contradictions among messages.Tables 24 and 25 present the message-list and hint-list cases, respectively.Source: §D.2 · p. 22, Table 24 · p. 22, Table 25 · p. 23

D.3 Case Study on Questions and Answers

  • The QA case studies cover simple, conditional, comparative, aggregative, post-processing, and noisy questions.The paper characterizes them as single-hop, multi-hop, or questions with added noise, depending on type.Source: §D.3 · pp. 24–26
  • A simple QA is constructed by rewriting one hint into a question, with its hint message as the retrieval target.The answer is the hint’s value.Source: §D.3 · p. 24
  • The conditional example retrieves the boss’s work location from the message stating the boss has only a high-school education.The example’s answer is Shenzhen, Guangdong.Source: §D.3 · p. 24
  • The comparative example asks who is older, Yalin Zhao or Wei Zhang, using their stated ages.The example’s answer is Yalin Zhao.Source: §D.3 · p. 25
  • The aggregative example asks how many people have high-school education or below and gives four messages as retrieval targets.Its stated answer is two people.Source: §D.3 · p. 25
  • The noisy example adds unrelated conversational material before asking for the hobbies of the person working in Shenzhen.The answer is sports, and the retrieval target is messages [2] [3].Source: §D.3 · p. 26

Related papers

Loading 2409.20163v1…