Source-linked AI summary
KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions
Tingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang, Chenglong Li, Shuo Zhang, Sen Hu, Silin Wu, Qizhen Lan, Huacan Wang, Ronghao Chen
TL;DR
Existing long-horizon memory benchmarks do not directly measure person understanding, motivating KnowMe-Bench’s use of dense autobiographical evidence and time-anchored narrative reconstruction. Across evaluated systems, retrieval-oriented methods improve factual recall, while chronology-sensitive and interpretive reasoning remain fragile without explicit temporal provenance.
Problem
Existing long-horizon memory benchmarks leave gaps in directly measuring person understanding.
Method
KnowMe-Bench reconstructs autobiographical narratives into flashback-aware, time-aligned cognitive streams and evaluates evidence-grounded understanding across structured memory protocols.
Results
Retrieval-augmented and entity-tracking systems improve factual recall, but remain fragile on chronology-sensitive and longer-range interpretive reasoning.
Takeaways & Limitations
The benchmark distinguishes precise fact recall from temporally grounded personal reasoning, showing that personal memory systems require explicit temporal provenance beyond static fact storage.
Takeaways & Limitations
The benchmark relies on an LLM-as-a-Judge protocol validated by human experts and incurs the operational complexity of multi-agent generation and de-identification.
Abstract
from arXiv · showhide
Existing long-horizon memory benchmarks mostly use multi-turn dialogues or synthetic user histories, which makes retrieval performance an imperfect proxy for person understanding. We present \BenchName, a publicly releasable benchmark built from long-form autobiographical narratives, where actions, context, and inner thoughts provide dense evidence for inferring stable motivations and decision principles. \BenchName~reconstructs each narrative into a flashback-aware, time-anchored stream and evaluates models with evidence-linked questions spanning factual recall, subjective state attribution, and principle-level reasoning. Across diverse narrative sources, retrieval-augmented systems mainly improve factual accuracy, while errors persist on temporally grounded explanations and higher-level inferences, highlighting the need for memory mechanisms beyond retrieval. Our data is in \href{KnowMeBench}{https://github.com/QuantaAlpha/KnowMeBench}.
1 Introduction
KnowMe-Bench frames person understanding as evidence-grounded inference over lived experience, addressing gaps in retrieval-focused evaluation and sparse, decontextualized user histories. It uses autobiographical narratives, flashback-aware cognitive streams, and hierarchical evidence-linked evaluation to measure capabilities from factual recall to principle-level reasoning.
- Research gaps: Existing benchmarks often test retrieval and state tracking, but do not directly measure inference of motivations, principles, self-concepts, relationships, or affective triggers.
- Research gaps: Sparse dialogue, synthetic, and model-generated histories weaken links between observable actions and internal deliberation while flattening modality and temporal cues.
- Benchmark design: KnowMe-Bench uses autobiographical narratives that preserve external events alongside internal interpretation as dense evidence for person-model inference.
- Benchmark design: Its cognitive-stream reconstruction anchors experiences to timestamps and locations across observations, sounds, context, background knowledge, and inner monologue, while handling mnemonic flashbacks.
- Evaluation and contributions: The benchmark evaluates factual extraction, subjective state attribution, and decision or principle reasoning with explicit supporting events and expert-validated annotations.
- Evaluation and contributions: KnowMe-Bench is publicly releasable, built from approximately 4.7M tokens, and accompanied by baselines spanning prompting, retrieval augmentation, and external memory systems.
2 Related Work
Related benchmarks primarily evaluate memory through retrieval, state tracking, static personas, or linear timelines. KnowMe-Bench instead emphasizes flashback-aware temporal representation and evidence-grounded interpretive reasoning over autobiographical experience.
- Memory evaluation: Long-term memory evaluations have progressed from context-window tests to multi-turn consolidation and entity-state tracking, with newer datasets extending beyond conversational traces.
- Memory evaluation: These evaluations generally prioritize recall fidelity and longitudinal consistency over interpretive reasoning about personal experience.
- Person modeling: Persona-agent studies commonly use static profiles or psychometric templates, which do not capture context-dependent behavior evolving over time.
- Timeline construction: Traditional timeline generation assumes linear progression or simplified timestamps, making it insufficient for narratives containing flashbacks and mental time travel.
- Positioning: KnowMe-Bench differs through flashback-aware chronological representation and evidence-grounded interpretive evaluation rather than only conversational recall or state tracking.
3 Methodology
KnowMe-Bench transforms autobiographical narratives into auditable, flashback-aware cognitive streams through extractive segmentation, atomic narrative units, temporal realignment, and embodied narrative generation. Verification agents and human literary experts protect fidelity throughout the pipeline.
- Pipeline: The framework applies a four-stage multi-agent workflow—segmentation, atomic-unit extraction, timeline generation, and narrative generation—with Verify-and-Revise safeguards.
- Pipeline: Context-aware segmentation detects semantic boundaries instead of fixed-length chunks and preserves source text verbatim through index-based extraction.
- Atomic representation: Atomic Narrative Units are the smallest auditable experience carriers, organized by unique identifier, temporal anchor, mandatory location, and structured cognitive record.
- Atomic representation: The cognitive record contains five primitives: Action, Dialogue, Environment, Background, and Mind, supporting high-density micro-evidence and global retrieval.
- Temporal reconstruction: Mnestic realignment separates historical event content from present mnemonic triggers and uses stack-based alignment to reorder flashbacks chronologically while preserving psychological continuity.
- Narrative generation: An embodied decoder converts objective descriptors into first-person sensory experiences, after which literary experts validate outputs and route errors back for revision.
4 Evaluation Framework
The evaluation suite measures long-horizon person understanding across seven tasks organized into factual, logical, and insight-oriented levels. Its LLM-as-a-Judge protocol uses task-specific rubrics and demonstrates substantial agreement with expert consensus.
- Evaluation levels: Seven tasks are hierarchically organized across three cognitive levels, spanning factual retention through literary reasoning.
- Factual evaluation: Factual tasks test entity and temporal-detail retrieval, adversarial abstention, duration estimation, and chronological ordering in flashback-heavy timelines.
- Logical evaluation: Logical tasks assess semantic event ordering and whether models identify sensory or associative triggers linking present scenes to recalled memories.
- Insight evaluation: Insight tasks require reconciling external behavior with internal state and answering expert-curated questions about motives, identity, and enduring decision principles.
- Scoring protocol: The GPT-4o judge scores factual accuracy and precision, logical sequence and reasoning validity, and external-to-internal mapping under task-specific rubrics.
- Scoring protocol: κ > 0.75 agreement with expert consensus indicates that the rubric-based judge tracks expert grading conventions on a held-out subjective subset.
5 Experiments
Experiments evaluate KnowMe-Bench across narrative regimes, human references, and memory architectures. Retrieval-oriented systems mainly help factual tasks, while chronological memory and explicit temporal scaffolding better support chronology-sensitive and interpretive reasoning.
- Datasets: The corpus contains 4.7M tokens across three narrative regimes and 2,580 queries, covering flashbacks, causal tracking, and introspection.Dataset 1 uses My Struggle, Dataset 2 the Neapolitan Novels, and Dataset 3 In Search of Lost Time.
- Evaluation: The evaluation combines aligned cognitive streams, original source text when needed, and expert-scored model and human responses.Three experts answered questions across all evaluation levels, with outputs scored by the same blinded LLM-as-a-Judge pipeline.
- Results: Experts score 96.5/88.0/83.5 on Levels I/II/III, versus 75.4/62.5/22.6 for the best model, leaving the largest gap at Level III.The reported κ > 0.75 measures judge-versus-expert alignment on the subjective subset.
- Baselines: The protocol compares Base, Naive-RAG, Mem0, and MemOS across four inference backbones, with A-Mem added for Qwen3-32B.Naive RAG uses k = 50 dense retrieval; Mem0 stores structured entity states; MemOS stores chronological logs.
- Results: On the flashback-heavy Dataset 1, Mem0 raises T1/T2 but lowers T3 by 3.5 points, whereas MemOS improves T3/T4 by 10.4/10.8.The pattern indicates that chronological logging better handles recalled material than state-updating memory.
- Results: Across backbones, retrieval systems are strongest on fact-heavy tasks, while MemOS more reliably improves chronology-sensitive and interpretive tasks.A-Mem reaches 55.7/41.3 on T6/T7 but falls to 16.0/16.5 on T3/T4; DeepSeek-R1 and Gemini-3 Pro also improve under MemOS on T3–T5.
6 Conclusion
KnowMe-Bench shifts lifelong-companion evaluation from fact retrieval toward auditable, evidence-grounded person understanding using dense autobiographical narratives. Results show that retrieval improves factual recall but remains fragile on flashback-aware chronology and longer-range interpretation, motivating memory systems with stronger temporal grounding.
- Contribution: KnowMe-Bench uses high-density autobiographical narratives to preserve actions, inner thoughts, and environmental context as aligned evidence.The benchmark is designed to remain auditable at the level of narrative evidence.
- Findings: Retrieval-augmented and entity-tracking systems improve factual recall but remain fragile on flashback-aware chronology, trigger–event linkage, and interpretive reasoning.Without explicit temporal provenance, recalled past experience can be conflated with present state.
- Implications: The benchmark provides flashback-aware reconstruction, evidence-linked evaluation, and diagnostic comparisons across memory architectures.These components distinguish retrieval competence from person-model inference.
Limitations
The benchmark’s evaluation must address literary-analysis subjectivity while absorbing the cost and operational complexity of its generation and de-identification pipeline.
- Scope: The benchmark relies on an LLM-as-a-Judge protocol validated by human experts and requires a costly, operationally complex multi-agent generation and de-identification pipeline.This limitation concerns both subjective evaluation and dense autobiographical-data processing.
Ethical considerations
The paper states that it follows licenses and usage policies and reports no additional privacy or human-rights risks from the benchmark.
- Licensing: The authors state that the benchmark follows the licenses and usage policies of the open-source models and datasets used.The claim is presented as an ethics and compliance commitment.
- Privacy: The authors report that the benchmark introduces no additional risks regarding data privacy or human-rights violations.The paper also describes a privacy pipeline that removes personally identifiable information while preserving narrative structure.
A.1 Semantic Divergence Metric (δ)
The pipeline uses semantic divergence δ as a propositional-mismatch check and triggers revision when generated content exceeds module-specific thresholds. It then constructs, audits, aligns, subjectivizes, and tests narrative evidence through constrained modules.
- Semantic Divergence Metric: Semantic divergence δ measures propositional mismatch rather than merely vector distance, using extracted atomic facts from source and generated texts.A Validator Agent extracts entities, timestamps, and actions from both texts for comparison.
- Semantic Divergence Metric: δ combines omission and hallucination rates, with α = 0.4 and β = 0.6 so hallucinations receive the larger penalty.The weighting is intended to prevent corruption of the ground truth.
- Revision Control: When δ exceeds the acceptance threshold ϵ, the system feeds validator feedback to regeneration for up to kmax = 3 iterations before manual review.The default threshold is ϵ = 0.03, while Module B uses 0.05.
- Narrative Reconstruction: The preprocessing pipeline preserves narrative structure through verbatim boundary-based slicing, atomic narrative units, and stack-based mnestic realignment.Realignment distinguishes MAINTAIN, PUSH, POP, and TRANSIENT actions for chronological placement and flashback scope.
- Narrative Reconstruction: Aligned units are instantiated as first-person experience only when all present fields are covered without unsupported emotional or descriptive additions.The instantiation gate rejects outputs when δ > 0.03 or embellishment is detected.
- Evaluation Generation: The benchmark generates retrieval and abstention questions with explicit spatiotemporal constraints, minimal evidence identifiers, and fixed ABSTAIN answers for deliberately false relations.Retrieval questions require unique answers, while adversarial questions preserve valid entities but distort their relations.
E Additional Experimental Results
Supplementary results provide per-dataset and backbone-specific views of the evaluation. The radar plots compare the same memory variants across backbones and reproduce the main task-level pattern.
- Supplementary Views: The appendix contains primary-backbone breakdowns used for dataset-level analysis and backbone-specific radar plots covering the full evaluation set.These views were moved from the main paper for space.
- Backbone Comparisons: Figure 4 compares Base, Naive RAG, Mem0, and MemOS across four backbone families, with A-Mem additionally evaluated on Qwen3-32B.The radar plots provide backbone-specific overall evaluation views.
- Backbone Comparisons: Retrieval-heavy variants help T1–T2, whereas MemOS yields broader gains on T3–T5 across the displayed overall results.The figure is described as showing the same pattern as Table 2.
F Human Evaluation Details
Human evaluation uses trained expert annotators who answer from the reconstructed stream with access to the original narrative. Their responses are scored by the same blinded judge used for model outputs, whose agreement with expert consensus is substantial.
- Annotators: Three M.A.-level literature or linguistics experts completed an evidence-grounded reading and question-answering study and were compensated at a reasonable rate.The study was designed as expert annotation rather than open-ended crowd annotation.
- Evaluation Materials: Annotators viewed the reconstructed cognitive stream and could consult the original novel to retain global narrative context during evaluation.This provided access to both the benchmark representation and its underlying narrative evidence.
- Scoring Pipeline: A blinded LLM judge scored human and model answers under the same level-specific rubric, without knowing the answer source.Annotators answered only with claims supported by textual evidence.
- Reliability: κ > 0.75 indicates substantial alignment between the rubric-based judge and expert consensus on a held-out subjective subset.The paper treats judge–expert alignment as the relevant reliability measure rather than pairwise human–human grading agreement.