PaperlayerView exact revision on arXiv

Source-linked AI summary

MEME: Multi-entity & Evolving Memory Evaluation

Seokwon Jung, Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seong Joon Oh

arXiv:2605.12477v1cs.LGcs.CL

TL;DR

Existing memory benchmarks largely omit interdependent, evolving entity updates, limiting evaluation of dependency reasoning. MEME benchmarks six tasks across these dimensions and finds that no practical-cost system reliably solves dependency reasoning, while closure requires an impractical configuration.

  • Problem

    Existing memory benchmarks evaluate updates only for independent entities, missing dependency reasoning over interdependent, evolving knowledge.

  • Method

    MEME evaluates six memory tasks using controlled episodes generated from DAG-based knowledge graphs with explicit conditional rules across two domains.

  • Results

    Cascade averages 0.03 and Absence 0.01 across all six systems, while no practical-cost system reliably solves dependency reasoning.

  • Takeaways & Limitations

    Dependency reasoning remains unsolved at practical cost; closure emerges only with Opus 4.7 on a file-based substrate costing approximately 70× the baseline.

  • Takeaways & Limitations

    Findings are bounded by two hand-crafted knowledge graphs, LLM-generated dialogues, 100 episodes, and limited-scope ablations.

Abstract

from arXiv · show

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 as its internal LLM partially closes the gap, but at ~70x the baseline cost, indicating closure currently depends on configurations that are not practical at scale. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/.

1 Introduction

MEME addresses a blind spot in memory benchmarks by evaluating stateful, interdependent knowledge across six tasks organized by entity scope and temporal dynamics. Its controlled DAG-based dataset provides verifiable ground truth for diagnosing where memory systems fail and where closure emerges.

  • MEME benchmark: MEME defines six tasks across entity scope and temporal dynamics: Exact Recall, Aggregation, Tracking, Deletion, Cascade, and Absence.The taxonomy spans single- versus multi-entity operations and static versus evolving information.
  • Motivation: Existing benchmarks evaluate updates for independent entities, missing dependency reasoning when upstream changes alter dependent facts.This leaves a blind spot in handling stateful, interdependent knowledge.
  • MEME benchmark: Cascade infers unstated changes from dependency rules, Absence recognizes when prior answers become untrustworthy, and Deletion verifies that removed facts are no longer reported.No existing benchmark scores these task types.
  • Dataset and diagnosis: MEME generates episodes from a DAG-based knowledge graph with explicit conditional rules across Personal Life and Software Project domains.The DAG structure makes gold answers verifiable by construction, while in-context validation confirms the tasks are solvable in principle.

2 Related Work

Prior work spans three LLM memory-system paradigms and benchmarks persistent memory across sessions, but existing tasks remain isolated and single-entity. MEME addresses this evaluation gap by targeting multi-entity and evolving memory.

  • Memory architectures: Three memory paradigms are raw retrieval, LLM-processed memory, and structured memory architectures.Raw retrieval preserves session utterances but is bounded by a fixed top-k window; LLM-processed systems extract or restructure information during ingestion.
  • Memory benchmarks: Stateless probes measure attention-window limits within one input, whereas multi-session benchmarks evaluate persistent memory across sessions.RULER and NoLiMa are stateless probes; LoCoMo, LongMemEval, MemBench, and MemoryAgentBench evaluate memory across sessions.
  • Memory benchmarks: Existing multi-session tasks evaluate evolving memory through updates, abstention, aggregation, and selective forgetting, but remain isolated and single-entity.LoCoMo tests static preferences, while LongMemEval, MemBench, and MemoryAgentBench extend evaluation to evolving memory.

3 MEME

MEME operationalizes the entity-scope × temporal-dynamics framework as six memory-intensive tasks and evaluates them through a DAG-based dataset-generation pipeline. The benchmark spans two domains and 100 episodes, with dependency-task answers computed by propagating updates through domain knowledge graphs.

  • Task design: Six tasks map the entity-scope × temporal-dynamics space, selecting representative memory-intensive operations while excluding easier variants covered by existing benchmarks.The tasks span static and evolving entities, including Exact Recall, Aggregation, Tracking, Deletion, Cascade, and Absence.
  • Task design: MEME covers Exact Recall, Aggregation, Tracking, Deletion, Cascade, and Absence, testing verbatim reproduction, retrieval coverage, revision histories, post-removal state, and dependency reasoning.Cascade infers dependent changes, whereas Absence tests when no answer is derivable from available rules.
  • Dataset construction: 100 evaluation episodes span Personal Life and Software Project domains, generated by sampling entity subsets, assigning tasks, verbalizing facts, and interleaving filler conversations.The dataset contains 50 episodes per domain and 694 post-change evaluation questions across the six task types.
  • Dataset construction: Each domain uses a Directed Acyclic Graph G = (V, E, P, Φ) whose conditional rules specify how descendant values depend on parent entities.The graphs encode entities, dependency edges, value pools, and rules such as changing medication after a health-condition update.
  • Dependency reasoning: For Cascade and Absence, gold answers are unstated in the conversations and computed by propagating upstream updates through the dependency graph, including multi-hop chains.Absence yields “Uncertain” when no applicable rule derives a dependent value.

4 Experiments

Across six memory systems, dependency reasoning is the central failure: Cascade and Absence remain near the floor despite adequate static-task performance. Prompt, retrieval, noise, and answering-LLM interventions largely fail, while only MD-flat with Claude Opus 4.7 narrows the gap at roughly 70× baseline cost.

  • Main results: 0.03 Cascade and 0.01 Absence average accuracy across six systems, below every static-task average and consistent across all three memory paradigms.MD-flat is the best system overall at 0.42, but dependency reasoning remains unreliable across N=5 samplings.
  • Main results: 0.02 is the Multi-Evolving cell after crossing both axes, while entity scope and temporal change independently reduce mean accuracy to 0.31 and 0.28.Each evaluation axis individually drops mean accuracy by approximately 0.30, showing that neither is redundant.
  • Main results: $0.16/ep in-context inference exceeds most memory systems’ $0.00–$0.04/ep, although in-context reaches Overall 0.36 and MD-flat reaches 0.42.Memory systems therefore become more cost-efficient as query volume grows.
  • Interventions: Prompt optimization, deeper retrieval, reduced filler noise, and a stronger answering LLM do not close the dependency-reasoning gap.Cascade remains near zero across retrieval depths, near the floor under prompt and noise changes, and averages 0.03 →0.02 when switching to Claude Sonnet 4.
  • Interventions: Cascade 0.32 and Absence 0.59 are reached only by MD-flat with Claude Opus 4.7, while the configuration runs at ∼70× the gpt-4.1-mini baseline.Closure requires both a capable internal LLM and a substrate that preserves contingency wording; the configuration also degrades Exact Recall and Tracking.

5 Conclusion

MEME shows that memory systems retain dependency rules and change events but fail to retrieve them correctly, leaving no practical-cost solution for dependency reasoning. Near-term workarounds rely on careful logging and templates, while longer-term progress requires architectures that propagate updates through dependent facts.

  • 5 Conclusion: MEME evaluates Cascade and Absence as dependency-reasoning tasks that test whether evolving context updates related facts and signals unresolved uncertainty.These task types were not covered by prior benchmarks.
  • 5 Conclusion: None of the practical-cost systems solves dependency reasoning: retrieval either ranks the pre-change value above the change event or fails to surface the event.Although dependency rules and change events are written and retained, the answering LLM reports the pre-change value.
  • 5 Conclusion: For dependency-heavy workloads, no practical-cost option exists today; near-term workarounds write dependency rules into conversation logs and surface change events through predefined templates.Existing systems suffice for retrieval-heavy or static workloads, whereas future progress requires architectures that natively propagate updates through dependent facts.

6 Limitations · A Operational Costs

The study’s findings are bounded by narrow, synthetic evaluation settings and limited scale. Its operational-cost analysis defines token accounting across pipeline stages and compares nominal per-episode costs under baseline and internal-LLM configurations.

  • 6 Limitations: The dataset contains two hand-crafted knowledge graphs, limiting evidence about generalization to broader domains or crowd-sourced graphs.The graphs cover Personal Life and Software Project.
  • 6 Limitations: LLM-generated dialogues constrain realism because they were not collected from real users.
  • 6 Limitations: 100 episodes with ∼35K-token contexts may not reveal patterns arising from longer contexts or larger sample sizes.
  • A Operational Costs: Ingest, Retrieve, and Answer separately count LLM calls for storing sessions, fetching context, and generating user-facing responses.In and Out columns distinguish prompt and completion tokens.
  • A Operational Costs: $15/$75 per 1M tokens is the listed Opus 4.7 input/output price, versus $0.40/$1.60 for gpt-4.1-mini.
  • A Operational Costs: The baseline cost table evaluates all systems end-to-end on gpt-4.1-mini, omitting negligible embedding API costs of ∼$0.0007/episode.Karpathy Wiki’s native query pipeline places retrieval and answer-generation costs in Retrieve.
  • A Operational Costs: The internal-LLM ablation compares three systems across four internal LLMs on a 20-episode subset while holding the answering model at Sonnet 4.Sonnet 4 answer-call tokens are included in the reported cost per episode.

B Dataset Construction Details … B.3 Example Generated Session

MEME uses hand-crafted domain graphs and controlled episode generation to evaluate six memory tasks across structured dependencies and stateless entity operations. Episodes are constructed with constrained value pools, dependency-specific targets, regeneration checks, and an illustrated self-chat session encoding five gold facts.

  • B.1 Knowledge Graph Structure: Each domain uses a hand-crafted knowledge graph G = (V, E, P) that is reused across all generated episodes.The graph structure summary appears in Table 7(a).
  • B.1 Knowledge Graph Structure: The Personal Life graph contains 39 entities across 9 categories, with 5 rotation roots and 11 dependency patterns.The roots are residence_location, employer, health_condition, relationship_status, and school.
  • B.1 Knowledge Graph Structure: The Software Project graph contains 51 entities across 6 categories, with 5 roots and 6 dependency patterns using manually authored fictitious values.Example values include Veltrion for framework and Crysthene DB for database.
  • B.2 Entity Values and Episode Generation: Entity values are sampled from manually curated pools of 5–15 distinguishable, semantically appropriate values, with root pools sized at 6–12.The complete value pools are provided in the released dataset files.
  • B.2 Entity Values and Episode Generation: Episode generation cycles through all 5 roots, constructs dependency chains outward, and selects Cascade and Absence targets among 1-hop and 2-hop descendants.Each hop level can contribute one Cascade target with an if-then replacement rule and one Absence target without a replacement value, yielding up to 4 dependency targets.
  • B.2 Entity Values and Episode Generation: Stateless tasks use non-chain entities: exact recall targets long complex values, deletion targets a personally meaningful orphan, and aggregation uses curated related-entity triples.The exact-recall entities are life_philosophy and error_log.
  • B.2 Entity Values and Episode Generation: Values are sampled uniformly under consistency constraints; Cascade after-values follow declared rules, Absence after-values become ⊥, and incomplete episodes are discarded and regenerated.Episodes must contain all 6 required task types, with up to 20 retries using a different root.
  • B.3 Example Generated Session: Figure 6 illustrates a Personal Life Episode 1 Fact Introduction session with root health_condition and five encoded gold facts.The facts are hobby = pottery, vehicle = Zyvanta Sedan, health_condition = lactose intolerance, sleep_pattern = varies by shift, and media_consumption = The Quorath Codex (book).

B.4 Haystack Assembly and Filler Filtering … E Prompt Optimization Details

MEME constructs filtered, interleaved haystacks from domain-specific corpora and evaluates memory systems under their native configurations and shared prompting procedures. Prompt optimization modifies ingestion and retrieval prompts, but its effects include unchanged or degraded performance and a drop in Absence accuracy to 0.00.

  • B.4 Haystack Assembly and Filler Filtering: 1,130 Personal Life and 9,135 Software Project sessions remain after length filtering from raw pools of 17,034 and 29,502 sessions, respectively.Fillers are drawn from LongMemEval-S and ShareGPT, then filtered against gold-fact vocabularies using retrieval and GPT-4o-mini conflict judgments.
  • B.4 Haystack Assembly and Filler Filtering: Personal Life filtering is dominated by CONTRADICTION flags, whereas Software Project filtering is dominated by ALTERNATIVE flags.Flag counts apply to gold-fact–filler pairs, and one filler may trigger multiple flags.
  • B.4 Haystack Assembly and Filler Filtering: 32K-token episodes interleave fillers and evidence sessions, allocating roughly B/N filler tokens across N gaps.A representative Personal Life episode contains 20 filler and 5 evidence sessions.
  • C Memory System Configuration: Off-the-shelf systems run with library defaults, preserving differences in top-k, chunking, retrieval mix, and internal-LLM logic.MD-flat is the benchmark authors’ minimal baseline, while each episode uses an isolated workspace.
  • C Memory System Configuration: The six systems span sparse and dense retrieval, fact memory, temporal graphs, file-based memory, and compiled knowledge articles.BM25 and text-embedding-3-small have no internal LLM; Mem0, Graphiti, MD-flat, and Karpathy Wiki use internal LLM-driven processing.
  • D.1 Verbalization Prompts: Conversation generation converts first-person fact seeds to third-person descriptions, then uses alternating User and Assistant LLMs to verbalize natural multi-turn exchanges.An annotation pass identifies which user turn conveyed each fact, followed by a separate Gemini 2.5 Flash semantic audit.
  • D Prompts: All systems share one answer prompt, while GPT-4o at temperature 0 uses task-specific judges for tracking, aggregation, exact recall, deletion, cascade, and absence.Aggregation and Tracking additionally receive partial credit based on present target values or correctly ordered history values.
  • E.1 Setup and Configuration: SIMBA optimizes only ingest and retrieve prompts for MD-flat, Mem0, Graphiti, and Karpathy Wiki using 10 training and 10 held-out test episodes.The answer prompt remains frozen because Claude Sonnet 4 lacks the OpenAI seed parameter needed for noise suppression.

F Noise Robustness Details · G Repeated-Run Stability

Increasing filler from 32K to 128K tokens degrades overall accuracy across MD-flat, Mem0, and text-embedding-3-small, while Cascade and Absence remain near zero across noise conditions. Repeated-run evaluation uses five identical trials on a 10-episode subset to test sampling stability and calibrate the noise floor for SIMBA.

  • F Noise Robustness Details: 40 episodes: The noise evaluation uses the same 20 PL and 20 SW episodes for each of three systems and three filler conditions.Deletion, Cascade, and Absence use the same trivial-pass filtering convention as the main results table.
  • F Noise Robustness Details: 0.45 to 0.36: MD-flat’s overall accuracy declines as filler increases from 32K to 128K.Mem0 declines from 0.26 to 0.22, while text-embedding-3-small declines from 0.23 to 0.16.
  • F Noise Robustness Details: −0.20: MD-flat’s Aggregation and Tracking scores each decline by this amount from 32K to 128K filler.Its degradation is broader than the retrieval-task-concentrated declines reported for Mem0 and text-embedding-3-small.
  • F Noise Robustness Details: 0.40 versus 0.45: MD-flat performs worse without filler than with 32K filler.With a short memory file, gpt-4.1-mini strips timestamps and entity labels, returning bare bullet lists that affect the answering LLM’s strict context-only behavior.
  • F Noise Robustness Details: 0.08 →0.23 →0.51: Raw retrieval’s Absence trivial-pass rate rises sharply across no-filler, 32K, and 128K conditions.The filter excludes passes where the system failed the pre-change state check, indicating it never knew the original value.
  • G Repeated-Run Stability: N=5: Two configurations are rerun under five identical trials on a 10-episode subset.The repeated runs assess whether single-run main-table results are sampling artifacts and calibrate the noise floor for the SIMBA prompt-optimization comparison.

G.1 Main-table baseline … I Per-stage Cascade Traces for the Remaining Systems

Across repeated trials, the dependency-reasoning gap remains stable and system rankings largely persist, while prompt optimization and per-stage traces expose distinct failure modes rather than resolving Cascade errors.

  • G.1 Main-table baseline: N=5 trials reran ingestion, retrieval, and answering for four nondeterministic systems on the 10-episode main-table subset.The configuration used gpt-4.1-mini internally and Claude Sonnet 4 for answering, including the original run plus four sampling trials.
  • G.1 Main-table baseline: Cascade SD ≤0.03 and Absence SD ≤0.04 across every system, demonstrating that the dependency gap is robust to sampling.Per-system Overall accuracy changes by at most 0.02 SD, preserving the main-table system ranking.
  • G.1 Main-table baseline: Tracking SD 0.15 is the largest task-level variance, occurring for Karpathy Wiki because its agentic query loop makes multiple internal LLM calls per question.Other systems use a fixed number of LLM calls and show smaller variance.
  • G.2 SIMBA prompt-optimization (MD-flat): +0.25 Overall is the largest baseline-versus-optimized change, observed for MD-flat in the N=5 SIMBA prompt-optimization re-evaluation.For Mem0, Graphiti, and Karpathy Wiki, the single-seed baseline-versus-optimized change is ≤0.02 Overall.
  • H Answering LLM Swap: Per-System Breakdown: Table 16 reports per-system accuracy after replacing gpt-4.1-mini with Claude Sonnet 4 as the answering LLM across six systems and 100 episodes.Internal LLMs remain gpt-4.1-mini, with trivial-pass filtering applied to Cascade, Absence, and Deletion.
  • I Per-stage Cascade Traces for the Remaining Systems: All four traced systems encode the rule and pre-change value, yet fail at different downstream stages and ultimately report the pre-change value, Hyunwoo Nam.The trace uses the same Cascade episode, sw_033, for Mem0, MD-flat, BM25, and text-embedding-3-small.

J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep) · K MD-flat Internal-LLM Mechanisms

The top-k sweep separates Cascade and Absence failures into retrieval misses and answering failures, showing that reasoning remains a major bottleneck even when dependency evidence is retrieved. The following section traces MD-flat’s internal-LLM storage mechanisms behind these outcomes.

  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): Both the dependency rule and upstream change event are checked to distinguish answering failures from retrieval failures.A failure is classified as answering when both are retrieved but propagation fails; missing either is a retrieval failure.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): In episode sw_033, the correct recipient changes to James Lee, but all four systems answer the pre-change value, Hyunwoo Nam.The episode changes the team lead, triggering the stated rule that the weekly report recipient becomes James Lee.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): Figure 30 shows BM25 and MD-flat with gpt-4.1-mini dropping the change event before retrieval, whereas text-embedding-3-small and Mem0 retrieve it but still answer with the pre-change value.The trace follows each system through encoding, maintenance, retrieval, and the answering LLM.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): Gold-fact metadata identifies the rule session and canonical change session for each Cascade and Absence task.The rule is marked by dependency metadata, while the change session contains the root-change fact for the cascade source.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): 100% of failures retrieve the rule across all k values, leaving change-event misses as the retrieval-side loss.At higher k, the trade-off is between missing the change event and failing to answer despite retrieving the relevant evidence.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): 86% and 83% of Absence failures at k=20 and k=40, respectively, are answering failures.These cases retrieve the relevant evidence but produce a definite answer when the system should abstain; the remainder are change-event misses.
  • J Retrieval vs. Reasoning Bottleneck on Cascade and Absence (Top-k Sweep): 55% of Cascade failures at both k=20 and k=40 are change-event misses, while 45% are answering failures.For the answering failures, both the rule and change session are retrieved, but the answering LLM does not produce the propagated value.
  • K MD-flat Internal-LLM Mechanisms: The MD-flat internal-LLM ablation is analyzed by tracing per-LLM internal-storage mechanisms behind the Cascade and Absence outcomes.This section connects those mechanisms to the internal-LLM ablation reported in Table 4.

K.1 MD-flat with gpt-5: ingestion compresses memory and erases prior entries

On MD-flat, gpt-5 detects Change+Delete sessions as update triggers but compresses memory on every write, shrinking stored context and often erasing facts needed for dependency reasoning. This produces apparently correct Absence answers by defaulting to missing-information responses rather than retrieving the relevant dependency evidence.

  • Write behavior: 95–100% write rate: gpt-5, GLM-5.1, and Opus 4.7 treat Change+Delete sessions as memory-update triggers.gpt-4.1-mini almost always uses append_memory.
  • Write behavior: ∼1,800 chars: gpt-5 compresses on every write, cumulatively shrinking pre-event memory.The passage identifies this shrinkage as gpt-5’s compression strategy.
  • Dependency reasoning: Trigger detection alone does not yield dependency reasoning because gpt-5’s compressed memory often lacks the original entity value and dependency rule.Retrieval consequently returns “(no relevant facts),” and the answering LLM defaults to “I don’t have that information,” matching the Absence gold of “uncertain.”
  • Concrete failure case: 4,440-char memory file: in Personal Life Episode 7, gpt-5 replaces 32 entries with a 236-char write containing only four new entries.The prior commute fact is deleted along with 31 other entries, leaving no relevant fact for the subsequent Absence retrieval.

K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation · L In-Context Ceiling: Gold Facts Only

Opus 4.7 closes dependency-reasoning gaps in MD-flat memory by storing explicit contingencies and propagating upstream changes, but this comes with substantial recall, tracking, and cost penalties. A gold-facts in-context ceiling confirms that the tasks are solvable when retrieval is perfect.

  • K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation: Opus 4.7 reorganizes MD-flat memory into topical sections and records dependency rules as explicit Contingency entries.This separates dependency rules from the dependent facts they govern.
  • K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation: For Cascade, Opus writes resolved 1-hop and 2-hop values into memory, allowing the answering LLM to retrieve the gold answers directly.The example resolves dietary restriction to no alcohol and exercise routine to yoga 2x/week at Crysthene Pool.
  • K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation: For Absence, Opus removes stale deployment-derived facts and marks them for re-confirmation, producing “I don’t have that information” and matching the uncertain gold answer.The removed facts include logs drain, Docker image, CI config, Dockerfile, and deploy command.
  • K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation: 0.00 to 0.32 Cascade and 0.07 to 0.59 Absence are the MD-flat jumps attributed to Opus’s explicit contingencies and active propagation.Other internal LLMs either leave stale facts, or in gpt-5’s case produce accidental Absence matches without genuine Cascade propagation.
  • K.2 MD-flat with Opus 4.7: explicit contingencies and active dependency propagation: 0.95 to 0.60 Exact Recall and 0.80 to 0.20 Tracking are the reported costs of Opus’s hierarchical reorganization on the same 20-episode subset.The per-episode cost is roughly 70 times the gpt-4.1-mini baseline.
  • L In-Context Ceiling: Gold Facts Only: The gold-facts in-context ceiling feeds only task-relevant gold facts without filler directly to the answering LLM and confirms that the tasks are solvable in principle with perfect retrieval.It uses no memory system and no filler.
Loading 2605.12477v1…