PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends

Chaoqun Zhan, Qiang Zhou, Guannan Li, Zhenqiang Huang, Qianjin Wang

arXiv:2608.13883v1cs.AI

TL;DR

Agent-memory benchmarks have limited evidence about whether memory supports interdependent, multi-session task completion rather than post-hoc recall. This study compares four memory backends across five MemoryArena domains under matched evaluation conditions and finds that MemoryLake leads observed SR in three domains, with a 20.5% versus 13.6% five-domain macro-average.

  • Problem

    Existing agent-memory benchmarks emphasize information recall rather than whether remembered information supports later actions and multi-step task completion.

  • Method

    The study compares MemoryLake, Mem0, vector RAG, and long context across five MemoryArena domains under a matched agent framework, model alias, task samples, and scoring code.

  • Results

    MemoryLake had the highest observed SR in mathematics, physics, and progressive retrieval, with a five-domain macro-average of 20.5% versus 13.6% for the best comparator.

  • Takeaways & Limitations

    The findings support a workload-dependent view of memory backends and an observed lead for MemoryLake among the four evaluated systems on the shared sets.

  • Takeaways & Limitations

    The reported rankings are point estimates without paired significance tests, so they do not establish statistically significant differences.

Abstract

from arXiv · show

Most agent-memory benchmarks test post-hoc recall, whereas MemoryArena evaluates whether memory supports interdependent, multi-session task completion. We compare MemoryLake, a structured multi-track memory backend, with Mem0, text-embedding-3-small vector RAG, and a long-context control across all five MemoryArena domains. The systems share the same agent framework, requested gpt-5-mini model alias, task samples, and scoring code; the memory integration is the intentionally changed component. Because each backend bundles write, retrieval, consolidation, budgeting, and prompt-assembly choices, the study is a matched system-level comparison, not a representation-only ablation or a cost-matched experiment. On the shared evaluation sets, MemoryLake has the highest observed success rate (SR) in mathematics (9/40), physics (12/20), and progressive retrieval (4/20). Every system has zero SR in travel planning, and web shopping yields a single bundle-level success (long context, 1/150); MemoryLake ranks third on both the travel soft process score and shopping step match. Following MemoryArena's suite-level convention, a post-hoc equal-weight average over the five SRs is 20.5% for MemoryLake versus 13.6% for the best comparator. These are point estimates: sample sizes are modest, confidence intervals overlap, and we do not report paired significance tests. A separate MemoryLake-only run over all 221 progressive queries yields a failure-counted SR of 26.7% (59/221) and is not a baseline comparison. The results support a workload-dependent view of memory backends and an observed lead among the four evaluated systems on the shared sets; they do not establish benchmark-wide state of the art or a causal advantage of representation structure.

1 Introduction

MemoryArena evaluates whether agent memory supports interdependent, multi-session task completion rather than only post-hoc recall. This study presents a matched system-level comparison of MemoryLake with three controls across all five domains, emphasizing workload-dependent performance and bounded claims.

  • Benchmark setting: MemoryArena uses a closed Memory–Agent–Environment loop with ordered, interdependent subtasks whose earlier information can be required later.The benchmark separates process-level completion from end-to-end success.
  • Compared systems: MemoryLake is a structured multi-track backend maintaining confirmed conclusions, supporting evidence, and reusable experience under different presence policies.It is compared with Mem0, naive vector-chunk RAG using text-embedding-3-small, and a same-model long-context configuration.
  • Study design: The study compares four memory configurations across all five MemoryArena domains under a matched agent framework, model alias, task sample, and evaluator.The design is a system-level comparison with exact sample sizes and denominators.
  • Analysis: The analysis reports per-domain SR, the benchmark’s equal-weight suite-level average SR, and process metrics while interpreting rankings across different workloads.The workload categories emphasize exact conclusion reuse, broad verbatim replay, constraint tracking, or aggregation of intermediate findings.
  • Claim scope: The evidence supports only a bounded claim about observed SR on the shared sets, not statistical significance, benchmark-wide state of the art, resource-matched superiority, or causal effects of a representation mechanism.The paper also identifies sampling, judge dependence, artifact scope, and benchmark-release ambiguities as reproducibility concerns.

2 Related Work

Prior benchmarks largely separate memory recall from consequential action, while MemoryArena evaluates their interaction across interdependent sessions. MemoryLake extends structured-memory systems with parallel tracks and distinct presence policies, alongside related decision-centric and skill-oriented approaches.

  • Recall-oriented memory benchmarks: Recall-oriented benchmarks test whether stored information can be recovered, but provide limited or absent environment feedback and cross-subtask action dependencies.LoCoMo and LongMemEval focus on long-conversation question answering, while MemoryAgentBench and MemoryBench add incremental interaction and continual learning.
  • Action-oriented agent benchmarks: Action-oriented benchmarks evaluate navigation, tool use, shopping, and software repair, but usually use independent instances without persistent cross-session memory transfer.Examples include WebArena, WebShop, and SWE-bench.
  • Memory–action coupled evaluation: MemoryArena combines memory, action, and environment feedback across interdependent sessions, probing procedural reuse, preference tracking, item compatibility, and finding aggregation.Its domains cover formal reasoning, travel planning, bundled shopping, and progressive web search.
  • Memory systems: Existing memory systems include extractive, trajectory-based, operating-system-style, graph-structured, reasoning, and skill abstractions; MemoryLake uses parallel tracks with different presence policies.This places MemoryLake within the structured-memory family while distinguishing it from a single homogeneous store.
  • Memory systems: DeMem and MSCE are closely related, but their published values are not same-protocol baselines because DeMem uses different task and evaluation configurations, while MSCE does not evaluateDeMem frames memory as a decision-centric rate–distortion problem; MSCE organizes grounded traces, procedural policies, and declarative cognition and promotes evidence-backed policies into callable skills.

3 Evaluation Setup

The evaluation compares four complete memory configurations under a matched agent framework, using shared models, task IDs, scoring, and cross-system evaluation sets. It covers five domains with task-specific datasets and metrics, while treating suite-level averages and rank statements as descriptive point estimates.

  • System comparison: Four systems share the MemoryArena framework, gpt-5-mini alias, task IDs, and official scoring scripts; only the memory integration is intentionally changed.MemoryLake also uses gpt-5-mini for internal backend operations, without a larger or auxiliary model.
  • System comparison: The comparison evaluates complete backend configurations, including writing, retrieval, consolidation, budgeting, prompt assembly, fallback behavior, and resource use.It is neither a representation-only ablation nor token-, latency-, or cost-matched.
  • Evaluation domains: The suite covers formal mathematics, formal physics, group travel planning, bundled web shopping, and progressive retrieval using shared cross-system sets.The common comparison uses only IDs shared by all four systems.
  • Evaluation domains: 40 mathematics papers contain 354 ordered subproblems, while progressive retrieval compares 20 fixed queries and 142 slots from 221 released queries.The progressive subset is a deterministic proportional stratified sample selected before scoring and shared by all systems.
  • Scoring and uncertainty: The five-domain SR average is post hoc and descriptive, and rank statements are point estimates rather than claims of statistical significance.Reported intervals are marginal, not paired, and the study does not report exact McNemar tests or clustered paired bootstrap intervals.

4 MemoryLake: Structured Multi-Track Memory

MemoryLake uses a structured multi-track memory pipeline that writes completed-subtask trajectories and assembles bounded memory context before subsequent subtasks. Its publicly identified models are standard components, while storage and policy heuristics remain proprietary and mechanism-level explanations are unablated hypotheses.

  • Memory interface: MemoryLake writes each completed subtask’s trajectory and outcome, then prepends an assembled memory context before the next subtask using one generic configuration across all five domains.No task-specific tuning is reported.
  • Implementation: All internal language-model operations use the requested gpt-5-mini alias, while dense retrieval uses the publicly available BAAI bge-m3 encoder in dense-only mode.No larger or auxiliary language model is used inside the backend.
  • Memory tracks: Three tracks separate confirmed conclusions prioritized for later presence, supporting evidence and procedural traces retrieved on demand, and reusable problem-solving experience consolidated across subtasks.The design hypothesis assigns different presence policies to these content types, but the experiment does not isolate them.
  • Memory assembly: At recall, MemoryLake filters or compacts redundant material under a bounded context budget and returns empty memory context rather than blocking the task if its internal service fails.Not all historical content is retained verbatim.
  • Reproducibility: Storage layout, indexing, consolidation, and assembly heuristics remain proprietary, so released artifacts support score verification but not full independent backend reproduction.The paper distinguishes score verification from backend reproducibility.

5 Results

MemoryLake has the highest observed success rate in mathematics, physics, and progressive retrieval, while travel and shopping show little end-to-end separation. Its descriptive shared-set macro-average is 20.5%, but these point estimates do not establish a benchmark-wide or causal lead.

  • Travel planning and web shopping: 0 SR occurs for every system in travel, while long context has the highest travel SPS at 53.1% and MemoryLake ranks third at 43.0%.In shopping, long context alone succeeds at bundle level with 1/150; MemoryLake ranks third in step match at 29.6%.
  • Suite-level summary: 20.5% macro-average SR is 6.9 percentage points above the best comparator’s 13.6% on the shared evaluation sets.The post-hoc equal-weight average summarizes five domain SRs and is descriptive rather than benchmark-wide.
  • Physics: 12/20 physics SR is MemoryLake’s largest observed margin, versus 9/20 for long context.The task requires exact reuse of intermediate derivations, but overlapping marginal intervals and no mechanism ablation limit interpretation.
  • Mathematics: 9/40 mathematics SR gives MemoryLake the highest observed point estimate, versus 8/40 for Mem0.Final SR ranges from 12.5% to 22.5%, while mathematical process scores range from 27.6% to 29.5%.
  • Progressive retrieval: 4/20 progressive-retrieval SR is highest for MemoryLake, versus 3/20 for Mem0 and 2/20 for each other system.Mem0’s 8.9% all-slots PS came from a capped re-run; its unmodified run scored 0/20 SR and 1.9% all-slots PS.
  • Additional progressive run: 26.7% failure-counted SR (59/221) is MemoryLake’s separate all-query progressive result, not a comparison with the 20-query baselines.The run reports a 95% Wilson interval of [21.3, 32.9] and counts incomplete runs, including input-length failures, as failures.

6 Discussion and Limitations

MemoryLake’s observed lead is workload dependent: it leads formal reasoning and progressive retrieval, while long context leads travel and shopping process metrics. The 20.5% versus 13.6% macro-average is descriptive, but subset coverage, absent paired inference, confounded mechanisms, resource differences, and reproducibility limits constrain interpretation.

  • Observed results: MemoryLake leads formal reasoning and progressive retrieval, while long context leads travel and shopping process metrics.The pattern argues against a universally ordered category of memory systems and motivates reporting process metrics alongside final success.
  • Observed results: 20.5% versus 13.6%: MemoryLake’s macro-average SR exceeds the best comparator on the shared sets.This is the strongest single descriptive statement supported by the current tables, bounded by subset design and incomplete baseline coverage.
  • Statistical limitations: Overlapping marginal Wilson intervals and absent paired tests prevent statistical-win claims.The math margin over Mem0, progressive margin over Mem0, and physics margin over long context are individual instances rather than paired-inference results.
  • Coverage limitations: Travel uses 30 of 270 groups and progressive comparison uses 20 of 221 queries, so the macro-average mixes subset and full-domain runs.A 50-bundle shopping sensitivity check also reorders step-match leaders relative to the complete set.
  • Causal limitations: The backend swap and embedder mismatch prevent attributing differences specifically to representation or other mechanisms.MemoryLake uses bge-m3 while Vector RAG uses text-embedding-3-small; mechanism claims require targeted, embedder-matched ablations.
  • Reproducibility and scope: The comparison is not efficiency-matched, and protocol artifacts limit reproducibility and benchmark-wide claims.Unmatched costs, a 35-versus-30 search-iteration discrepancy, a hard-coded 32,000-token output budget, unresolved model snapshots, and limited baseline coverage all constrain interpretation.

7 Conclusion

Across matched evaluations, MemoryLake led the four evaluated systems on mathematics, physics, and progressive retrieval, with a higher suite-level average but no benchmark-wide state-of-the-art claim. The results support a workload-dependent view of memory backends, with stronger claims requiring broader and more rigorous evaluation.

  • Comparative results: MemoryLake achieved the highest observed SR in mathematics (9/40), physics (12/20), and progressive retrieval (4/20) across the matched shared sets.The comparison used the same agent framework, requested model alias, task IDs, and scoring code.
  • Comparative results: Every system had zero end-to-end SR in travel, while long context recorded the only shopping success (1/150); MemoryLake ranked third on both travel SPS and shopping step match.
  • Aggregate result: 20.5% was MemoryLake’s post-hoc equal-weight average across five SRs, versus 13.6% for the best comparator.These are observed end-to-end results among four evaluated systems on shared sets, not evidence of benchmark-wide state of the art.
  • Interpretation and limitations: The findings support a workload-dependent view: MemoryLake is most promising for exact conclusions or intermediate-result aggregation, whereas long context remains competitive for broad, high-fidelity replay.Stronger leadership claims require full commonset runs, a pinned model snapshot, paired inference, resource logging, and a complete public result artifact.

Competing Interests

All authors are affiliated with MemoryLake, creating a conflict of interest because the study compares its commercial system with third-party baselines. Official evaluators and reporting of negative results provide safeguards, but mechanism explanations remain hypotheses and rankings are protocol-bounded.

  • All authors are affiliated with MemoryLake, a commercial memory system, while the compared systems are third-party baselines.
  • The study uses the benchmark’s official evaluators and reports negative as well as positive results, but these safeguards do not remove the conflict of interest.
  • Mechanism explanations should be treated as hypotheses, and system rankings are bounded by the stated protocol.

Code and Data Availability

The MemoryArena environments and evaluators are publicly available, and the companion repository provides the study materials and artifacts needed to reproduce aggregate results from stored per-instance outputs.

  • The MemoryArena environments and evaluators are available from the benchmark authors.
  • The companion repository contains the method description, evaluation settings, exact sample manifests, per-instance scored outputs, and aggregation scripts.
  • The released artifacts support recomputing the aggregate values in Tables 3–5 from the stored per-instance results.

A Alignment with the MemoryArena v1 Release

The section frames the paper’s quantitative claims as within-experiment findings based on shared samples and evaluators, rather than a re-evaluation of MemoryArena’s contribution.

  • The quantitative claims are within-experiment claims based on shared samples and evaluators, not a re-evaluation of MemoryArena’s contribution.

B Scoring and Verification Notes

The evaluation uses domain-specific scoring rules and fixed cross-system denominators, with an independent aggregation program serving as an internal consistency check rather than an independent replication. Available artifacts include transcripts and evaluator I/O, but complete judge transcripts and inter-judge agreement statistics remain future additions.

  • Scoring: 40 mathematics papers, 20 physics papers, 30 travel groups, 150 shopping bundles, and 20 progressive-retrieval queries define the cross-system SR denominators.These denominators accompany domain-specific evaluation procedures.
  • Scoring: Mathematics and physics use LLM-based equivalence judging, travel uses deterministic slot matching, shopping uses exact ASIN matching, and progressive retrieval combines LLM judging with slot-level PS/SR aggregation.The progressive-retrieval aggregation is described in Section 3.4 and is not shipped by the benchmark.
  • Verification: The independent aggregation program is an internal consistency check, with exact agreement on deterministic tasks supporting consistent denominator and formula implementation.It is applied to the authors’ stored results.
  • Verification: Sub-one-percentage-point differences on judged tasks are not treated as independent replication because parsing, rounding, and judge-output handling are undocumented.The released artifact preserves raw results, but the source does not establish why all differences occur.
  • Artifacts and limitations: Agent transcripts and full evaluator input–output are included where available, while complete verbatim judge transcripts and inter-judge agreement statistics remain future additions.The limitation concerns the completeness of released evaluation evidence.
Loading 2608.13883v1…