Source-linked AI summary
ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
Wenjie Zhang, Zhichao Shi, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Xiaoning Song
TL;DR
Personal agents must use interaction history to identify needed external knowledge and apply it to a user’s circumstances, a joint evaluation gap in existing benchmarks. ColaBench traces these dependencies, and MemKB links state summaries and historical memories to knowledge retrieval. MemKB achieves 86.25% Accuracy versus 79.94% for Mem0+Hybrid using the same memories, while requiring fewer average interaction rounds.
Problem
Existing benchmarks cover memory recall, personalized retrieval, and memory-guided action, but joint evaluation of history-guided knowledge acquisition and application remains underexplored.
Source: §1 · pp. 1–2Method
ColaBench links personal conditions, their disclosures, document exposure, and answer requirements in an Evidence Graph, while MemKB uses state summaries and memory–document links to guide retrieval.
Source: §1 · p. 2Results
86.25% Accuracy versus 79.94% for Mem0+Hybrid using the same memories, a gain of 6.30 percentage points; average interaction rounds fell from 4.44 to 4.07.
Source: §1 · p. 2Takeaways & Limitations
ColaBench provides traceable tasks for studying how memory guides knowledge acquisition and rule application, while MemKB improves accuracy with fewer interaction rounds.
Source: §8 · p. 8Generalization to new users and other domains, live Web environments, and the reliability of automated evaluation remain to be evaluated; task answers have not been clinically validated.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
1 Introduction
ColaBench evaluates how personal history guides both knowledge acquisition and rule application, while MemKB links historical conditions to knowledge retrieval. The benchmark and method address a joint evaluation gap and show higher accuracy with fewer interaction rounds than a memory-matched baseline.
1 Introduction
- Existing benchmarks cover memory recall, personalized retrieval, or memory-guided action, but joint evaluation of history-guided knowledge acquisition and application remains underexplored [9] [16] [15].The paper identifies unsupported personal assumptions and rule misapplication as obstacles even when relevant evidence is available.Source: §1 · p. 2
- ColaBench’s Evidence Graph links personal Facts to disclosures, document exposure, and answer requirements, supporting traceable evaluation and diagnosis across 349 Items, 10 Profiles, and 13,033 documents.The graph supports distinguishing errors involving missing personal conditions, missing knowledge, or the use of evidence in an answer.Source: §1 · p. 2
- MemKB uses state summaries to inform queries and judgments, and memory–document links to combine history-associated knowledge with direct semantic and keyword retrieval.The answering agent combines retrieved memories and document evidence to assess which rules apply to the user’s circumstances.Source: §1 · p. 2
- 86.25% Accuracy versus 79.94% for Mem0+Hybrid using the same memories (+6.30 percentage points), with average interaction rounds falling from 4.44 to 4.07.Gains were larger on tasks requiring multiple historical Facts: 9.35 percentage points versus 4.96 on single-Fact tasks.Source: §1 · p. 2
2 Related Work
Related work spans long-term memory, personalized conversational search, and evidence-level diagnosis. ColaBench studies knowledge search, evidence reading, and rule application while tracing how user state supports required conclusions.
Long-Term Memory and Task Execution
- LoCoMo and LongMemEval evaluate cross-session memory and temporal reasoning [9] [16], while Mem2ActBench and MemoryArena examine memory-guided tool use and interdependent actions [15].ColaBench extends this line to knowledge search, evidence reading, and rule application, tracing how recovered user state supports conclusions.Source: §2 · p. 2
Personalized Conversational Search
Evidence-Level Diagnosis
- HippoRAG integrates evidence across passages [7], ALCE evaluates supporting citations [4], and RAGChecker diagnoses retrieval coverage and evidence use [13].ColaBench connects answer requirements to personal-condition sources and rule applicability conditions to examine whether evidence was obtained and correctly applied.Source: §2 · pp. 2–3
3 Evidence-Graph-Based Construction
ColaBench uses an Evidence Graph whose document, profile, fact, topic, turn, and item nodes capture links and information boundaries throughout staged construction. Disclosure and exposure records constrain what can inform later tasks.
3 Evidence-Graph-Based Construction
- The Evidence Graph defines task dependencies by linking personal conditions to their disclosures and answer requirements to personal and knowledge evidence.The graph records document exposure and guides knowledge exploration during construction.Source: §3 · p. 3
3.1 Task Definition
- The task setting x = (q, H, K; M) comprises the current request, preceding public history, frozen knowledge base, and evaluation metrics.Tasks examine how agents use personal conditions from history to acquire and apply external knowledge.Source: §3.1 · p. 3
3.2 Stage 0: Building the Document Layer
- The document layer begins with 11,069 screened-source documents, then adds graph relations grounded in knowledge relationships and documents found through web search.These documents and links supply knowledge and exploration paths for Profile generation and Item construction.Source: §3.2 · p. 3
3.3 Stage I: Grounding Profiles and Facts
- A Director plans scenarios using retrieved documents to create personal Facts and Topics, while an independent reviewer checks consistency, relevance, completeness, and source support.Accepted content is added to the Evidence Graph.Source: §3.3 · p. 3
3.4 Stage II: Growing Public Interaction Traces
- A user Simulator gradually discloses personal conditions as topics unfold, and accepted conversation turns record Fact disclosures and document exposure for later construction.The Assistant responds using the public conversation and retrieved knowledge.Source: §3.4 · p. 3
3.5 Stage III: Constructing Items
- An initial Item starts from disclosed conditions and new knowledge whose application requires them; a history-removal test checks that completion depends on the interaction history.Accepted Items link required Facts, historical Turns, and supporting Documents.Source: §3.5 · p. 3
3.6 Item Enhancement and Validation
- Enhancement develops multi-condition decision tasks by reorganizing condition dependencies or replacing the supporting document, followed by independent history-dependence and evidence review.The public interaction history remains fixed during enhancement.Source: §3.6 · pp. 3–4, Figure 1 · p. 4
3.7 Dataset Overview
- 10 Profiles, 349 Items, and 13,033 documents comprise ColaBench; multi-Fact tasks cite at least two annotated Facts, and multi-session tasks draw evidence from at least two sessions.Appendix A.2 provides further structural statistics.Source: §3.7 · p. 4
4 MemKB: Memory-Linked Retrieval
MemKB uses historical state and memory–document links to guide retrieval, then returns memory, state, and document evidence to support user-specific rule application.
4 MemKB: Memory-Linked Retrieval
- MemKB connects the knowledge a request requires with how retrieved rules apply to the user, using a state summary before query generation and memory–document links.It fuses linked documents with documents retrieved directly for the current query.Source: §4 · p. 4
4.1 Memory Cards and Document Links
- Memory cards link surviving memory IDs to source messages and historical Read documents, then add unexposed semantic neighbors found by querying the full knowledge base.Reference-document annotations are withheld from the builder during neighbor discovery.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
4.3 From User State to Rule Application
- The complete state bank accompanies each search, while the answering agent compares rule conditions with user state and memory records and preserves uncertainty when source or event status is unspecified.The prompt requires attribution of the speaker, actor, and event status in memory records.Source: §4.1 · p. 5, §4.3 · p. 6
4.2 Memory-Linked Knowledge Retrieval
- A query retrieves relevant memory cards, and round-robin expansion distributes candidate slots across their linked documents before equal-weight RRF fuses memory-linked, semantic, and keyword results.Each returned document includes a relevant snippet, document ID, and line range.Source: §4.2 · p. 5
5 Experimental Setup
The experiments compare memory systems on a shared benchmark setup and assess both answer correctness and evidence access. Evaluation also measures interaction cost and multiple dimensions of answer quality.
5.1 Methods and Experimental Settings
- Mem0, Graphiti, and A-MEM are paired with Hybrid retrieval, while MemKB uses historical document-access records, memory-linked retrieval, and a state summary before the first query.The memory methods are built from the same public user and assistant messages; Hybrid combines semantic and keyword retrieval over the knowledge base through RRF.Source: §5.1 · p. 6
- The main comparison uses all 349 tasks across 10 Profiles with a shared frozen knowledge base and up to four tool-call rounds before final answering.Search returns five documents by default, up to 20; Read returns at most 100 lines and 3,000 tokens per call.Source: §5.1 · p. 6
5.2 Evaluation Metrics
- Accuracy requires correct, complete responses to requested subquestions, including relevant personal conditions, applicability requirements, and limitations.Evaluation combines the public history, reference answer, requirements, supporting documents, and each method’s actual Search and Read returns.Source: §5.2 · p. 6
- Read Recall measures whether annotated new-support documents have nonempty Read returns before the final answer, while search snippets and memory may also support answers directly.Each task has one reference document, so document coverage is analyzed alongside Accuracy.Source: §5.2 · p. 6
- Avg. Interaction Rounds counts mean agent rounds, including the final answering round; five additional blinded answer-quality dimensions are scored from 1 to 5.Each dimension is assessed in a separate GLM-5.3 Flash call using the same reference inputs and candidate-visible evidence as Accuracy.Source: §5.2 · p. 6
6 Results and Analysis
MemKB outperforms memory-retrieval baselines on answer accuracy, with larger gains on tasks combining multiple historical conditions. Analyses of shared document access, component ablations, cases, and failures examine how memory organization and rule application relate to those outcomes.
6.1 Answer Quality and Knowledge Access
- 86.25% Accuracy on 349 tasks exceeds Mem0+Hybrid by 6.30 percentage points despite similar Read Recall, while MemKB uses fewer interaction rounds.MemKB also exceeds A-MEM+Hybrid by 4.87 percentage points and uses 0.35 and 0.37 fewer rounds than A-MEM+Hybrid and Mem0+Hybrid, respectively.Source: §6.1 · p. 6
- On 111 tasks, MemKB scores higher across all five answer-quality dimensions, with the largest gains in Condition Coverage and Rule Application at +0.207 each.On 16 tasks MemKB answered correctly while Mem0+Hybrid did not, mean Rule Application gain is 1.563 points.Source: §6.1 · p. 6
6.2 Combining Personal Information
- 9.35 percentage points is MemKB’s Accuracy gain on multi-Fact tasks, versus 4.96 points on single-Fact tasks; gains are also larger for tasks citing at least three interaction pairs.The history-pair gains are 8.57 points for at least three pairs and 5.33 points for one or two; 91 multi-Fact tasks also cite at least three pairs. Under common composition weights, gains are 4.87 and 14.41 points on 316 tasks; adding cohort yields 5.86 and 12.24 points for single- and multi-Fact tasks on 275 tasks.Source: §6.2 · p. 7, Figure 3 · p. 7
6.3 Shared Access and Answer Quality
- When both methods receive the reference document, Accuracy is 90.44% for MemKB versus 86.03% for Mem0+Hybrid; when both read it, scores are 90.74% and 87.96%, respectively.The Both exposed and Both read subsets contain 136 and 108 tasks, respectively; the figure defines these access conditions and reports paired gains.Source: §6.3 · p. 7, Figure 3 · p. 7
6.4 Case Studies: Rule Application
- In paired cases, MemKB preserves cancer history and distinguishes past from current treatment, while another case shows both methods assigning opioid use without confirmation.The latter illustrates that a prior discussion of a drug class does not establish personal use; unknown premises require conditional branches.Source: §6.4 · p. 7
6.5 State Summaries and Document Links
- Removing state summaries and document links reduces Accuracy by 6.31 and 10.81 percentage points, respectively.Without summaries, first-search hit rate falls from 36.94% to 22.52% and average rounds rise from 4.27 to 4.49. Without links, Read Recall falls from 42.34% to 36.94% even as full-interaction search hit rate rises from 46.85% to 61.26%; with links, MemKB reads the reference more often and completes more tasks correctly.Source: §6.5 · pp. 7–8, Table 6 · p. 8
6.6 Error Analysis
- Personal Assumptions are the largest error category; other failures include missing decision-relevant evidence and misapplying a visible rule.Examples include an unsupported opioid classification, an omitted contraindication absent from retrieved context, and a recommendation that conflicts with retrieved heat-risk guidance.Source: §6.6 · p. 8, Figure 4 · p. 8
7 Discussion and Limitations
The evaluation uses ten Profiles, with component ablations on three; generalization to new users, other domains, and live Web settings remains unevaluated, and task answers lack clinical validation.
7 Discussion and Limitations
- Method development and comparison use ten Profiles, while component ablations cover three selected Profiles.Source: §7 · p. 8
- Generalization to new users, other domains, and live Web environments remains to be evaluated.Source: §7 · p. 8
- Automated evaluation reliability requires independent data and human calibration, and task answers have not been clinically validated.Source: §7 · p. 8
- The frozen health and everyday-life corpus makes document versions, sources, and reading scope reproducible.Source: §7 · p. 8
8 Conclusion
ColaBench connects answer requirements to historical and document evidence to evaluate memory-guided knowledge acquisition and application. Its frozen knowledge base draws on public APIs, XML, web pages, and open textbooks.
8 Conclusion
- ColaBench evaluates memory-guided knowledge acquisition and application through an Evidence Graph linking answer requirements to historical and document evidence.Source: §8 · p. 8
- MemKB improves task accuracy with fewer interaction rounds, and ablations support contributions from state summaries and document links.Source: §8 · p. 8
- Together, ColaBench and MemKB connect memory evaluation to the knowledge agents acquire and the user-specific judgments they make.Source: §8 · p. 8
A.1 Corpus Sources
- The frozen knowledge base extracts text from public APIs, XML files, web pages, and open textbooks, preserving titles, body text, and source URLs.Documents are organized by entry, chapter, or drug-label field; web additions are deduplicated by body text.Source: §A.1 · p. 9
A.2 Task Annotations
The appendix describes task annotation, evidence construction, and validation procedures that keep personal conditions grounded in public history and assess whether new evidence supports answers.
A.2 Task Annotations
- 349 tasks are annotated as Single-Fact or multi-Fact according to whether they cite one Fact or at least two Facts.A Fact can contain several related personal conditions; 1,015 answer requirements are used to evaluate Accuracy and analyze task composition.Source: §A.2 · p. 9
A.3 History and Evidence Construction
- Public user statements establish personal conditions before the current request, while histories retain tool arguments and document navigation information.Methods receive current requests and information through their supplied states, memories, and tool outputs; private Facts, reference answers, requirements, and reference-document annotations are withheld.Source: §A.3 · pp. 9–10
- Task construction excludes previously exposed documents and exact duplicate sources, and records each search event before a document is read.Document-identity checks and semantic validation determine whether required knowledge was already available in history.Source: §A.3 · p. 10
A.4 Acceptance and Evidence Mapping
- A history-removal test distinguishes answerable-without-history tasks from history-dependent tasks and cases with insufficient evidence.Final validation checks natural task progression, sourced personal conditions, historical knowledge insufficiency, new-evidence support, answer completeness, and safety.Source: §A.4 · p. 10
- Each answer requirement links its text to user statements, personal conditions, knowledge rules, and a required judgment or action.Personal evidence traces to user messages and knowledge evidence to documents; unknown premises use conditional branches, with programmatic integrity checks and model assessment of semantic support.Source: §A.4 · p. 10
- Refinement jointly updates questions, answers, requirements, and evidence mappings while preserving public history and personal conditions.Multi-condition enhancement combines at least two disclosed conditions around the task and requires dependence on multiple conditions; drafts failing validation are excluded or revised and re-evaluated.Source: §A.4 · p. 10
B.1 Memory and Retrieval Implementations
The appendix specifies memory and retrieval implementations, ablation conditions, and paired estimation. MemKB’s Accuracy gain over Mem0+Hybrid is estimated at 6.30 percentage points across the 349-task comparison.
B.1 Memory and Retrieval Implementations
- Mem0+Hybrid and MemKB share the same memories; MemKB retains valid historical document links and offline-neighbor links, but not unresolved historical Read references.Source: §B.1 · p. 10
- Graphiti stores complete interaction pairs as episodes and retrieves relational facts with built-in RRF; A-MEM uses note analysis, neighbor linking, memory evolution, and retrieval expansion.Memory stores are built from the same 599 public interaction pairs, retaining message roles and sources.Source: §B.1 · p. 10
- Memory baselines use separate memory_search and kb_search tools, whereas MemKB uses memkb_search; all methods share Read and the knowledge index.Source: §B.1 · p. 10
B.2 Model and Retrieval Configuration
- Offline discovery retrieves 20 semantic candidates per card and excludes documents previously exposed in history.Retrieved neighbors precede historical Read documents, and round-robin expansion deduplicates across cards.Source: §B.2 · p. 10
B.3 Component Ablations
- The state-summary ablation removes supplied state lists, while the document-link ablation clears linked documents but retains memory vectors and state summaries.The ablations cover all 111 retained tasks from three Profiles and use the same models, evaluation criteria, and interaction budgets as the full configuration.Source: §B.3 · p. 10
C.1 Paired Estimation
- 6.30 percentage points is MemKB’s Accuracy gain over Mem0+Hybrid, with a 95% confidence interval of [3.19, 9.09].The estimate uses paired results on all 349 tasks and a cluster bootstrap over ten Profiles with 20,000 replicates.Source: §C.1 · p. 10
- Table 9 summarizes the main-comparison configuration, with output limits applying per call and the agent budget applying per task.Source: Table 9 · p. 11
C.2 Task-Composition Sensitivity
The task-composition analysis reports positive multi-Fact gain differences whose confidence intervals span zero. Shared-background tasks call for different rules, and the case analyses illustrate errors involving personal assumptions, retrieval, and rule application.
C.2 Task-Composition Sensitivity
- 4.39, 9.54, and 6.39 percentage-point multi-Fact gain differences had 95% confidence intervals spanning zero in unadjusted, Profile/requirement-adjusted, and cohort-adjusted analyses.The analyses used common-support sets of 316 and 275 tasks; the intervals were exploratory and unadjusted for multiple comparisons.Source: §C.2 · p. 11
C.3 Shared Background across Tasks
- 60 of 98 Profile-and-Fact groups involved at least two reference documents, covering 311 tasks (89.1%); three shared-background tasks required different rules.The rules depended on the relative’s condition and the caregiver’s distance recorded in the history.Source: §C.3 · p. 11, Table 10 · p. 12
C.4 Case Evidence
- The HFC eligibility example shows caregiving role and co-residence each determine whether relevant rules apply.The example combines public statements about caregiving and living alone with the frozen Care Grants page.Source: §C.4 · p. 11
- In the treatment-history case, Mem0+Hybrid asserted no cancer treatment or radiation record, while MemKB kept the user’s actual treatment details conditional.Both methods read the same NICHD document, which included chemotherapy guidance and a rule for women undergoing radiation therapy.Source: §C.4 · p. 11
- A scalp-cooling search snippet omitted the cold-triggered migraine exclusion, while a neuropathy answer recommended warm foot baths despite a source listing them as a heat risk.In the neuropathy case, the agent read the document listing warm foot baths among heat risks.Source: §C.4 · pp. 11–12
D.1 Metrics and Scoring Inputs
The section defines how answers, retrieval, and evaluation inputs are scored, then specifies answer-quality dimensions and error categories. Evidence visibility is assessed using what the agent actually received, not evaluator-only material or unreturned candidates.
D.1 Metrics and Scoring Inputs
- Accuracy is the share of passing answers among all tasks in a comparison; Read Recall counts unique reference documents with nonempty Read returns, while search hit rates track reference-document presence in successful search responses.Search hit-rate denominators include all cohort tasks, treating tasks without successful searches as misses.Source: §D.1 · p. 12
D.2 Answer-Quality Dimensions
- Answer quality is assessed across State Fidelity, Condition Coverage, Evidence Grounding, Rule Application, and Answer Completeness.Judgments use candidate excerpts, source evidence, and a rationale.Source: §D.2 · p. 12
D.3 Error Categories and Evidence Visibility
- Primary error categories distinguish Personal Assumptions, Context Underuse, Rule Misapplication, Answer Omissions, and Other Issues; visibility records whether needed personal premises and rules were available in agent-visible evidence.Visibility can be Visible, Partly visible, Not visible, or Uncertain; no answer received the Uncertain label.Source: §D.3 · p. 12