PaperlayerView exact revision on arXiv

Source-linked AI summary

CloneMem: Benchmarking Long-Term Memory for AI Clones

Sen Hu, Zhiyu Zhang, Yuxiang Wei, Xueran Han, Zhenheng Tang, Huacan Wang, Ronghao Chen

arXiv:2601.07023v1cs.AI

TL;DR

AI Clone memory benchmarks have largely relied on fragmented conversational histories, limiting evaluation of continuous changes in people’s experiences, emotions, and opinions. CLONEMEM builds a hierarchical benchmark from non-conversational digital traces and evaluates evidence-grounded reasoning over evolving life trajectories. Experiments show that existing memory systems struggle, with simple retrieval often outperforming abstractive systems and models relying on generic narratives when evidence is underspecified.

  • Problem

    Existing benchmarks rely mainly on conversational histories and single-source data, which do not adequately evaluate how experiences, emotions, and opinions evolve over time.

  • Method

    CLONEMEM hierarchically generates longitudinal life traces and evaluates retrieval and reasoning over evidence-linked diaries, social posts, messages, and related artifacts.

  • Results

    Across the benchmark, simple flat retrieval often outperforms abstractive memory systems, while AI Clones struggle with precise trace grounding and generic narrative substitutions.

  • Takeaways & Limitations

    Effective AI Clone memory must preserve high-fidelity longitudinal evidence and model what brings about changes in an individual’s experiences, emotions, and opinions.

  • Takeaways & Limitations

    The dataset is synthetic, represents non-textual artifacts textually, and relies on LLM-as-a-judge evaluation that may introduce bias.

Abstract

from arXiv · show

AI Clones aim to simulate an individual's thoughts and behaviors to enable long-term, personalized interaction, placing stringent demands on memory systems to model experiences, emotions, and opinions over time. Existing memory benchmarks primarily rely on user-agent conversational histories, which are temporally fragmented and insufficient for capturing continuous life trajectories. We introduce CloneMem, a benchmark for evaluating longterm memory in AI Clone scenarios grounded in non-conversational digital traces, including diaries, social media posts, and emails, spanning one to three years. CloneMem adopts a hierarchical data construction framework to ensure longitudinal coherence and defines tasks that assess an agent's ability to track evolving personal states. Experiments show that current memory mechanisms struggle in this setting, highlighting open challenges for life-grounded personalized AI. Code and dataset are available at https://github.com/AvatarMemory/CloneMemBench

1 Introduction

AI Clones require memory grounded in evolving life experiences, emotions, and opinions, but existing conversation-based benchmarks capture only fragmented snapshots. CLONEMEM addresses this gap with non-conversational digital traces and tasks for tracking personal change over time.

  • Motivation: Existing memory benchmarks rely mainly on user–agent conversational histories that provide fragmented, episodic snapshots of people’s experiences.Such data make gradual psychological changes and everyday life beyond conversations difficult to track.
  • Motivation: Everyday diaries, social media posts, direct messages, and emails provide longitudinal traces for modeling experiences, emotional changes, and evolving opinions.These traces can support personalized interactions and a coherent digital proxy.
  • Contribution: CLONEMEM evaluates whether AI Clones can integrate non-conversational digital traces to track experiences, emotions, and opinions consistently over time.The benchmark extends memory evaluation beyond traditional conversation-based settings.
  • Contribution: The benchmark uses a top-down construction process that derives fine-grained daily traces from macro-level life arcs to preserve longitudinal coherence.The process models continuity in experiences, emotions, and opinions.
  • Contribution: Evaluation tasks test whether AI Clones can use non-conversational traces to track changes in an individual’s experiences, emotions, and opinions over time.Experiments indicate that existing memory systems face limitations in these AI Clone scenarios.

2 Related Work

Prior work studies agent memory, personalization, and long-context understanding, but typically assumes conversational or single-source data and largely stable personas. CLONEMEM instead evaluates how a person’s experiences, emotions, and opinions evolve across non-conversational digital life traces.

  • Agent Memory System: Agent memory systems address finite context windows through external storage, knowledge graphs, gist memories, evolving graphs, or self-organizing memory.Examples include MemGPT, HippoRAG, ReadAgent, Zep, Mem0, and A-MEM.
  • Personalization and Role Play: Personalization research has examined role adoption, predefined personality consistency, and personality development through ongoing interaction.Character-LLM, PersonaLLM, and AI Persona represent these directions.
  • Personalization and Role Play: Most existing personalization work assumes that a person’s persona remains largely unchanged, whereas CLONEMEM asks whether an AI Clone reflects change after new life events.This shifts attention from static persona consistency to longitudinal development.
  • Long-Term Memory Benchmarks: Long-term memory benchmarks evaluate multi-session memory, temporal reasoning, information updating, or performance under extremely long inputs.These benchmarks include LoCoMo, LongMemEval, MemBench, LongBench v2, BABILong, and RULER.
  • Long-Term Memory Benchmarks: Existing benchmarks typically use conversational histories or individual documents, making it difficult to capture evolving experiences, emotions, and opinions over time.CLONEMEM addresses this gap through non-conversational digital life traces.

3 Data Construction for CLONEMEM

CLONEMEM constructs longitudinal digital lives hierarchically, moving from personas and life arcs to phases, detailed events, evidence, and daily traces. Evidence-grounded filtering and review then produce temporally supported question–answer instances.

  • Hierarchical Event and Trace Generation: The construction process moves hierarchically from long-term life planning to fine-grained daily digital traces.This three-level organization links macro-level arcs, meso-level phases, and micro-level artifacts.
  • Persona and Macro-Level Life Arcs: A psychologically coherent persona and macro-level life arcs define long-term trajectories in experiences, emotions, and opinions.Major events provide coarse temporal milestones without fixing detailed outcomes.
  • Meso-Level Phase Generation: Meso-level phases connect major events to daily traces through persistent internal states that maintain emotional and attentional continuity.These states make changes reflect accumulated experiences rather than isolated signs.
  • Micro-Level Digital Trace Generation: Micro-level generation produces diaries, social media posts, and direct messages from detailed events, with evidence grounding the traces in underlying experiences and internal states.The pipeline organizes extracted evidence into arc-specific buckets for question–answer generation.
  • Evidence-Grounded QA Construction: Question–answer instances are tied to specific life arcs and localized temporal spans, and are generated only when evidence sufficiently supports the question category.Automated filtering removes approximately 10% of generated questions, followed by limited human review.

4 CLONEMEM Evaluation Benchmark

CLONEMEM evaluates long-term AI Clone memory through evidence-grounded reasoning over non-conversational digital traces. Its tasks span factual recall, temporal change, and higher-level reasoning across bilingual, very long-context data.

  • Benchmark Design: CLONEMEM tasks evaluate an AI Clone’s ability to comprehend and reason about an individual’s long-term, evolving life story.The benchmark differs from fact-oriented question answering over conversational histories.
  • Benchmark Design: Each question links digital traces to evidence units, defining evaluation as retrieval from traces followed by reasoning over the required evidence.Evidence units and traces have a many-to-many relationship.
  • Dataset: The benchmark provides bilingual English and Chinese data for 10 personas and approximately 5,000 question–answer pairs.Seven personas exceed 500k tokens, with some reaching 1M, while three are around 100k tokens.
  • Question Format: Questions are posed from the perspective of a close friend at a specific time and often require locating relevant information in long-term digital traces.CLONEMEM supports both free-text and multiple-choice formats.
  • Evaluation Tasks: Evaluation covers factual recall, temporal reasoning about changes and trajectories, and higher-level reasoning about causality, counterfactuals, abstraction, and unanswerable cases.These are organized as three levels of reasoning.

5 Experimental Setup

CLONEMEM evaluates memory systems over shared longitudinal digital traces using a standardized retrieval-and-generation pipeline. It compares memory architectures and measures retrieval, memory utility, and response quality.

  • All evaluated methods use the same long-term, non-conversational digital traces and unified pipeline for memory construction, retrieval, and response generation.
  • The experiments use LLaMA-3.1-8B and GPT-4o-mini as backbone models, with Contriever and text-embedding-3-small as embedding models.
  • At inference, systems retrieve k ∈{5, 10, 20} items by embedding similarity, then generate answers from the retrieved context.
  • The comparison includes Flat as a non-updatable retriever and A-Mem and Mem0 as updatable memory systems.
  • Evaluation covers retrieval, memory utility, and generation quality using Recall@K variants, Choice Accuracy, Memory Helpfulness Score, and QA Consistency Score.

6 Experiment Results

Across CLONEMEM experiments, simple flat retrieval is generally more reliable than consolidated memory systems, while performance depends on the backbone, retrieval depth, and retrieval-unit composition. The results show a recurring gap between semantic retrieval quality and precise trajectory-grounded reasoning.

  • Retrieval Performance: Flat retrieval consistently outperforms A-Mem and Mem0 across almost all retrieval metrics and both backbone models.Both backbones struggle particularly with Recall-All-Any, which requires evidence from every required evidence set.
  • QA Performance: The Oracle setting forms a clear upper bound across both backbones, indicating that remaining errors mainly arise from memory construction or retrieval.
  • QA Performance: Flat retrieval achieves the best or near-best memory helpfulness and recall for both backbones at every retrieval depth k.For LLaMA-3.1-8B, larger k can reduce QA consistency and Choice Accuracy, whereas GPT-4o-mini is more robust and A-Mem is often competitive for consistency at k=5/10.
  • Embedding and Extraction Model Choice: Text-embedding-3-small nearly doubles retrieval recall, but GPT-4o-mini paired with a weaker retriever still outperforms LLaMA-3.1-8B with a stronger retriever on consistency and accuracy.The results separate retrieval capacity from downstream reasoning capacity.
  • Impact of Retrieval Unit Composition: Extracted memories yield the highest semantic recall and helpfulness, while raw context alone yields the highest Choice Accuracy.This ablation exposes a validity–fidelity trade-off between summary-based semantic indexing and preserving detailed original traces.

7 Discussion: When Existing Memory Systems Fail for AI Clones

The discussion identifies a mismatch between abstraction-based memory and AI Clone demands for trace-level, state-aware evidence. Systems can retrieve relevant information yet generate plausible narratives or infer commitments that the record does not support.

  • Flat retrieval is the most reliable non-oracle baseline because AI-Clone queries require timestamps, phrasing, repetition, and cross-event linkage.Consolidation can improve semantic indexing while weakening alignment with the original evidence units.
  • When evidence is underspecified, models may replace the true mechanism of change with a fluent narrative template, producing emotionally plausible but causally wrong answers.
  • Dense activity traces can be misread as commitments, causing models to invent preferences or actions instead of answering “not specified” when the record is inconclusive.
  • The proposed direction is evidence-preserving memory that retains trace fidelity, represents internal-state transitions, and supports abstention through persistent-state modeling.

8 Conclusion

CLONEMEM evaluates AI Clones using non-conversational longitudinal traces to model evolving opinions, emotions, and experiences. The authors find that flat retrieval can outperform abstractive memory systems, while temporal compression and generic narratives undermine fidelity.

  • CLONEMEM evaluates AI Clones’ ability to model evolving opinions, emotions, and experiences from non-conversational longitudinal digital traces.
  • Simple flat retrieval often outperforms abstractive memory systems because compression can discard critical temporal cues.
  • AI Clones often rely on generic narrative templates and struggle to distinguish exploratory activities from commitments.
  • The findings motivate memory architectures that preserve trace fidelity while modeling longitudinal trajectories and the reasons behind change.

Limitations

CLONEMEM’s hierarchical construction pipeline progressively generates coherent personas, life trajectories, phases, events, evidence, digital traces, and trajectory-based questions. It combines rolling state updates with explicit evidence grounding to preserve temporal continuity.

  • Data construction: The four-stage pipeline refines abstract personality traits into temporally grounded traces and trajectory-based question–answer instances.
  • Stage I: Persona and Macro-Level Planning: Persona and macro-level planning establishes Big Five-based profiles, persistent relations, life arcs, and major events along a global timeline.
  • Stage II: Meso-Level Rolling Generation: Meso-level rolling generation decomposes major events into phases while carrying updated snapshots of experiences and emotional states across events.
  • Stage III: Micro-Level Event, Evidence, and Trace Generation: Micro-level generation expands phases into detailed events and explicit evidence that jointly ground diaries, social media posts, and direct messages.
  • Stage IV: Trajectory-Based Question-Answer Generation: Trajectory-based question generation aggregates evidence over life-arc-specific sliding windows to probe evolving experiences, emotions, and opinions.
  • Implementation: The benchmark uses Claude-4.5-Haiku for Stages I–III and Gemini-3-Flash for trajectory-based question–answer and multiple-choice construction.

B Evaluation Tasks

CLONEMEM evaluates long-term memory through questions grounded in non-conversational traces and asked from a close friend’s perspective at a specific time. Tasks span factual retrieval, temporal reasoning, inference, causality, counterfactuals, and recognizing missing evidence.

  • Task setup: CLONEMEM questions are posed from a close friend’s perspective and require integrating relevant information from long-term non-conversational traces.
  • Task taxonomy: The evaluation spans factual retrieval, temporal change and continuity, and higher-level reasoning over causality, alternatives, abstraction, and missing evidence.
  • Single-Point Factual Reasoning: Single-point factual reasoning tests retrieval of explicit information about a person’s state, activities, or opinions at a given time.
  • Temporal reasoning: Comparative reasoning and trajectory analysis require identifying change or stability between time points and characterizing evolution across longer periods.
  • Higher-level reasoning: Pattern, causal, and counterfactual reasoning examine recurring behavior, influences of earlier experiences, and alternative outcomes under different past choices.
  • Unanswerable Questions: Unanswerable questions test whether an AI Clone recognizes when records omit the queried outcome or issue and answers that it is not specified.

C Dataset Statistics

CLONEMEM reports dataset composition across personas, questions, languages, context lengths, semantic dimensions, question types, and media types. Its evaluation uses retrieval, semantic memory, and final-answer quality metrics.

  • Dataset overview: Table 2 summarizes the number of personas, questions, languages, and context-length settings in CLONEMEM.
  • Dataset composition: Figure 5 presents question distributions by semantic dimension and question type, alongside the media-type composition of digital traces.
  • Evaluation metrics: CLONEMEM evaluates media-level retrieval, semantic-level memory relevance and helpfulness, and QA truthfulness and memory dependence.

D.1.1 Media-Level Retrieval Metrics

CLONEMEM evaluates retrieval and answer quality when questions require multiple evidence units from longitudinal digital traces. It compares flat retrieval with updatable memory systems and measures both semantic coverage and personalized, truthful generation.

  • Retrieval metrics: A single question may require multiple media items spanning years, so CLONEMEM defines evidence sets and evaluates retrieval at several strictness levels.The metrics range from independent media-item recall to retrieving all items for all evidence sets.
  • Semantic retrieval evaluation: Memory Recall measures the proportion of ground-truth memory points semantically covered by retrieved traces, while Memory Helpfulness scores task-specific utility from 0 to 2.The helpfulness scale distinguishes conflicting information, partial evidence, and comprehensive evidence.
  • Generation metrics: Generation is evaluated for truthfulness and consistency, distinguishing hallucinated, generic, partial, and fully evidence-grounded answers.QA Consistency ranges from 0 to 3, with perfect answers using all relevant ground-truth evidence.
  • Generation metrics: QA Perfect Rate is the percentage of responses with a QA_consistency_score of 3, requiring truthful and exhaustive long-context recall.QA Hallucination Rate separately measures the percentage of responses that contradict or invent beyond the ground-truth memory.
  • Scoring rationale: The benchmark penalizes generic safe answers because an AI Clone must demonstrate detailed knowledge of an individual’s life trajectory.This scoring rationale treats generic correctness without personalized memory use as a failure of long-term memory.
  • Baselines: The experiments compare A-Mem and Mem0 with a non-updatable Flat retriever, which stores static extracted chunks and uses semantic similarity search without reconciliation or relational linking.A-Mem uses an evolving knowledge network, whereas Mem0 extracts and updates a streamlined set of high-level facts.

E Analysis by Question Type and Dimension

CLONEMEM reveals a mismatch between retrieval coverage and reliable life-trajectory reasoning. Models struggle with temporally distributed evidence, internal mechanisms of change, and distinguishing sustained exploration from stable commitment.

  • Retrieval by question type: Trajectory questions have the lowest retrieval recall because their evidence is distributed across time, whereas unanswerable and causal questions often retrieve salient related traces.Semantic cues can make retrieval appear successful even when the context does not support a definitive answer.
  • Retrieval versus reasoning: High retrieval recall does not guarantee accurate answers: flat retrieval performs weakest on unanswerable questions because models often fail to abstain when related content is present.Trajectory questions show the opposite pattern, with low recall but relatively stronger consistency once partial evidence is retrieved.
  • Counterfactual reasoning: Even when the decisive diary evidence is retrieved, models may replace persona-specific internal mechanisms with familiar external narratives in counterfactual reasoning.The documented error favors grades and counseling over the evidence linking divorce fear, mask breakdown, and reconnection.
  • Emotion reasoning: Models often confuse a surface cue with the mechanism of emotional change, treating another person’s words as the cause instead of tracking the belief shift toward vulnerability.The benchmark distinguishes increased awareness of a problem from the internal change that produces honest conversation and later behavior.
  • Emotion reasoning: Event-focused memory representations can retrieve interactions while failing to preserve the internal belief update that explains behavioral reversal.This allows fluent family-reconciliation narratives to replace the persona’s actual causal pivot.
  • Cognitive–action gap: Dense job-search activity can be misread as commitment, causing models to invent preferences or actions even when the persona remains undecided.The ground truth marks the case unanswerable because repeated exploration never specifies a target company.
  • Cognitive–action gap: Correct handling of exploration requires answering “not determined” rather than producing a plausible but unsupported next-step narrative.CLONEMEM identifies this as Exploration–Commitment Confusion and treats invented targets or actions as incorrect.
Loading 2601.07023v1…