PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

Wei-Chieh Huang, Weizhi Zhang, Yuchen Wu, Yankai Chen, Eric Hanchen Jiang, Wooseong Yang, Yiwei Yang, Henry Peng Zou, Hanrong Zhang, Ying Nian Wu, Haolun Wu, Kai-Wei Chang, Philip S. Yu, Xue Liu, Aylin Caliskan

arXiv:2608.15008v1cs.CL

TL;DR

Existing evaluations provide limited guidance on which memory substrate suits different operating regimes. This paper conducts a controlled cross-regime evaluation and finds that optimal substrates vary by task and history depth, making substrate routing necessary.

  • Problem

    Existing evaluations provide limited evidence about which memory substrate is preferable under different operating regimes.

  • Method

    The authors isolate memory substrate as the experimental variable in a shared harness spanning 11 methods, three backbones, and four benchmarks.

  • Results

    The optimal substrate reverses between long-form QA and agentic decision-making and shifts again with history depth.

  • Takeaways & Limitations

    Substrate routing is necessary, with heterogeneous memories composed and routed according to operating regime.

  • Takeaways & Limitations

    Two substrates deviate from their reference implementations to preserve comparability or accommodate unsupported kernels.

Abstract

from arXiv · show

Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate, namely the underlying medium in which memory is represented and stored, should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability introduces a further routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents. Code will be made available upon acceptance.

1 Introduction

Long-horizon agents require persistent memory, but existing evaluations provide little guidance on which memory substrate suits different operating regimes. This paper addresses that gap with a controlled, broad harness evaluation and finds that substrate choice should be regime- and horizon-aware.

  • Motivation: Long-horizon agents across coding, browsing, personal assistance, and scientific discovery require persistent memory across repeated interactions.These domains accumulate project context, browsing patterns, user preferences, or hypotheses over time.
  • Evaluation gap: Existing evaluations are concentrated and narrow: 62% use LoCoMo or LongMemEval, every system reports accuracy, and only 21% report any efficiency metric.Agentic tasks, broader metric coverage, and deployment-relevant efficiency remain underexamined.
  • Evaluation design: The study isolates memory substrate as the sole experimental variable across 11 methods, seven substrate families, three backbones, four benchmarks, and 26 metrics.The methods span external and internal memory under identical interaction histories.
  • Cross-regime findings: Optimal substrates reverse across regimes: structural graphs lead dialogue QA but are Pareto-dominated on agentic tasks, while refinement memories lead embodied planning but trail on QA.Retrieving more entries helps QA but hurts agentic decisions as attention shifts away from task context.
  • Scalability and design implication: Refinement memories scale gracefully, whereas structural and full-context substrates face long-horizon deployment limits, motivating fewer retrievals and deeper write-time distillation.The resulting design rule is to trade read breadth for write depth; no single substrate serves every regime.

2 Memory Substrates and Configuration

This section frames memory-substrate evaluation around configuration-specific benefits and trade-offs across task settings and operating regimes. It surveys eleven substrates organized into seven families, spanning increasingly structured external-memory designs.

  • Configuration Motivation: Memory configurations involve many tunable parameters and modular choices, motivating evaluation of regime-specific benefits and trade-offs rather than identifying one inherently superior method.This analysis is presented as an empirical prerequisite for systems that dynamically select substrates.
  • Substrate Overview: The study examines eleven memory substrates organized into seven families, including external methods spanning a spectrum of structures.The substrate overview is provided in Table 4 of Appendix B.
  • External Substrate Families: The external spectrum ranges from flat dense and sparse vector indices to gisted text records, linked structural stores, and hierarchical memory.Dense and sparse vectors require no LLM calls; Gist Index adds write-side gisting and read-side page selection; Evolving Notes and Dual-Level Graph introduce relational structure; Hierarchical Tree recursively clusters and summarizes memory.

3 Experiment Design

The study evaluates memory substrate families under a shared harness across user-centric and agent-centric benchmarks, asking how substrates affect performance and efficiency across operating regimes. Controlled retrieval-breadth and context-window ablations further separate retrieval effects from recency-aware update semantics.

  • Experimental scope: The harness holds backbone models, prompting, and auxiliary-LLM settings constant to isolate the memory substrate’s contribution to task performance and efficiency.It tests whether substrate effects remain stable across operating regimes.
  • Benchmarks: Four benchmarks span user-centric and agent-centric evaluation, including LoCoMo’s ten multi-session dialogues and 1,986 questions across five categories.MemoryAgentBench factors long-context memory into Accurate Retrieval, Long-Range Understanding, Test-Time Learning, and Conflict Resolution.
  • Models, inference, and metrics: QWEN3-8B, QWEN3-32B-AWQ, and GEMMA-4-26B-A4B-IT are tested on all four benchmarks using vLLM on 4×H200 GPUs.GPT-4O-MINI is fixed for auxiliary memory operations and LLM-as-a-judge evaluation, with 26 metrics instrumented per run.
  • Ablations: A top-k retrieval-breadth sweep uses k ∈{1, 2, 5, 10, 20} on LoCoMo and k ∈{1, 2, 3, 4, 5} on ALFWorld, alongside a no-memory baseline.The baseline uses the same template with an empty retrieved block, attributing quality differences to retrieval rather than scaffolding.
  • Ablations: The ablations hold substrate implementation, backbone, and prompt template fixed while testing retrieval breadth and context-window stress on MAB CR.Together, they distinguish retrieval-breadth advantages from recency-aware update semantics.

4 Results

Results show that memory-substrate performance is regime- and capability-dependent rather than universally dominated by one substrate. Broad or structural retrieval benefits factual and code-oriented tasks, while denoising and compact refinement better support embodied sequential decisions, with substantial cost and backbone interactions.

  • User-centric results: No single substrate wins every capability, and the best choice varies across capabilities and backbones.On LoCoMo and LME-S, M5 performs best across model bands, while M3 leads QWEN3-32B-AWQ and MAB results vary by memory policy.
  • User-centric results: M5 performs best across model bands on LoCoMo and LME-S, combining entity-level graph traversal with chunk-level vector search for long-dialogue fact retrieval.On MAB CR, M5 leads two of three models, consistent with explicit update semantics for recency-sensitive conflicts.
  • Efficiency and routing: M2 is consistently cheap, whereas M5 can be 10–100× slower and is worthwhile only when its structural mechanism matches the task.Cost separates otherwise competitive methods, making efficiency an additional substrate-selection axis.
  • Agent-centric results: On ALFWorld, M11 doubles NoMem on QWEN3-8B, while M7 reaches the peak TSR of 32.1% on QWEN3-32B-AWQ by distilling trajectories into compact strategies.These substrates denoise accumulated experience before it reaches the policy rather than exposing the full trajectory buffer.
  • Agent-centric results: On QWEN3-32B-AWQ ALFWorld, M2 scores 21.6 versus 22.4 for NoMem, while M7 reaches 32.1%; on QWEN3-8B, M2 scores 6.7 versus 5.7 for NoMem.Raw-text retrieval therefore depends on the backbone rather than being universally harmful for embodied planning.
  • Agent-centric results: On BigCodeBench-Hard, M5 leads QWEN3-8B and GEMMA-4 with 15.5% and 20.9% Pass@1, while M2 lifts Pass@1 above NoMem on every backbone.Retrieved code snippets extend the task prompt as reusable scaffolding, unlike retrieval in embodied planning.

5 Discussion

Discussion identifies distinct Pareto frontiers across QA and agentic regimes, shows that retrieval depth can help factual QA but harm action selection, and establishes scalability as a third routing axis. Together, these findings argue for multi-substrate or multi-agent memory systems composed by role rather than a single universally optimal substrate.

  • 5.1 Pareto frontiers: Structural and hierarchical memories occupy the QA frontier, while flat retrieval occupies the agentic frontier with M2 reaching +2.0% at 1.28× overhead on BCB-Hard.M5 wins at 10 to 30× baseline latency, whereas M6 has steady-state per-query cost comparable to Flat baselines after its one-off tree build.
  • 5.1 Pareto frontiers: Additional latency rarely buys quality monotonically: M4 at 234 s underperforms M2 at 7.3 s on LME-S, while M9 at 172 s reaches P4=0.29.On BCB-Hard, M5, M9, and M11 incur 3.7 to 8.0 times overhead while falling below NoMem by ∆Pass@1 between −1.4% and −4.1%.
  • 5.2 Retrieval depth and attention: Increasing retrieval depth raises P4 monotonically on LoCoMo but reduces task success on ALFWorld, where M7 drops from 32.1% at k=1 to ∼25% at k=5.The passage attributes the contrast to surplus context being passively ignorable for factual QA but harmful in sequential decision-making.
  • 5.2 Retrieval depth and attention: Growing k shifts attention from task Context into Retrieved content on both LoCoMo and ALFWorld, demonstrating universal attention dilution with task-dependent consequences.The probe measures last-prompt-token attention under M1 retrieval across System, Retrieved, task Context, and Cue regions.
  • 5.3 Scalability: At 6K-to-262K-token histories, M8 improves from 0.32 to 0.51 and M5 from 0.28 to 0.48 on MAB Conflict Resolution, making scalability a third routing axis.Longer histories provide more material for write-side abstraction and entity-relation updates to disambiguate which fact is current.
  • 5.4 Synthesis: No single substrate simultaneously serves long-history recall, action selection under attention pressure, and graceful long-horizon scaling; the proposed path is role-based multi-substrate or multi-agent memory.The passage assigns refinement substrates to curating abstractions and structural and text substrates to holding facts.

6 Related Work

Related work spans diverse external, internal, and hybrid memory substrates, while existing evaluations usually fix configurations and separate user-centric from agent-centric regimes. This work instead treats the substrate as the experimental variable and evaluates both regimes with deployment-relevant efficiency dimensions.

  • Memory-augmented LLM agents: Memory-augmented agents use external stores, internal stores, and hybrids, but most systems commit to a fixed configuration at design time.External stores include vector indices, text buffers, knowledge graphs, hierarchies, and distilled skill memories; internal stores include parametric updates and KV-cache activations.
  • Memory evaluation benchmarks and practices: The proposed harness isolates the memory substrate as the experimental variable and addresses both regime separation and deployment-critical evaluation gaps.The harness evaluates user-centric and agent-centric settings while incorporating dimensions such as write cost, latency, storage, and management overhead.
  • Memory evaluation benchmarks and practices: Existing benchmarks cover user-centric and agent-centric settings, but typically evaluate the two regimes in isolation and emphasize end-task accuracy.They often omit write cost, latency, storage, and management overhead, which are critical during deployment.

7 Conclusion · A Landscape of LLM Memory-Augmented Systems · A.1 Inclusion criteria and data collection

The paper evaluates agent memory substrates systematically and concludes that substrate choice must adapt to task and history depth. Its accompanying survey defines a structured corpus and inclusion framework for memory-augmented LLM systems.

  • 7 Conclusion: 11 substrates, three backbones, four benchmarks, and 26 metrics comprise the controlled harness evaluation of agent memory.The evaluation spans multiple substrate types and operating regimes.
  • 7 Conclusion: The optimal substrate reverses between long-form QA and agentic decision-making, then shifts again with history depth.These findings make substrate routing necessary rather than optional.
  • A Landscape of LLM Memory-Augmented Systems: 52 memory-augmented LLM systems released between early 2023 and April 2026 form the appendix’s survey collection.The survey focuses on how the current work evaluates systems rather than cataloguing individual mechanisms.
  • A.1 Inclusion criteria and data collection: Included papers propose a novel memory method with an associated substrate or architecture, evaluate at least one memory benchmark, and provide a public preprint.Survey, benchmark-only, and purely qualitative systems were excluded, while metadata and evaluation settings were independently verified against primary sources.
  • A.1 Inclusion criteria and data collection: Four evaluation dimensions are recorded for each system, including benchmarks grouped into six canonical categories and performance metrics such as accuracy and per-type breakdowns.The benchmark categories include LC, LME, MH, DS, DMR, and OTHER.
  • A.1 Inclusion criteria and data collection: Benchmark categories cover LoCoMo, LongMemEval, multi-hop QA, DialSim, DMR, and diverse other evaluations including MSC, Carecall, and MemoryAgentBench.The passage also lists additional datasets and custom evaluations under OTHER.

A.2 Aggregate statistics … B.2 Internal Substrates

The surveyed memory-agent literature is concentrated in a few benchmarks and backbones, with accuracy reporting far outpacing efficiency evaluation. The implementation harness spans eleven substrates, from external retrieval and structured stores to weight- and activation-based internal mechanisms.

  • A.2 Aggregate statistics: LoCoMo and LongMemEval account for 62% of adopted benchmark pairs, while adding DMR and DialSim raises the dialogue-centric share to 66%.LoCoMo contributes 29 of 76 pairs and LongMemEval 18; DMR and DialSim add three dialogue-centric pairs.
  • A.2 Aggregate statistics: 100% of systems report end-task accuracy, but only 21% report any efficiency metric, and no system reports management cost or compression ratio.Only LightMem reports four efficiency dimensions simultaneously, while Mem0 and Memoria each report two.
  • A.2 Aggregate statistics: 81% of systems use GPT-family backbones, and no system combines broad benchmark coverage with comprehensive efficiency reporting.Forty-two of 52 systems use GPT-family models; only two evaluate on at least three benchmarks, and no system occupies the ≥3-benchmark, ≥3-efficiency region.
  • A.3 Trends over time: The 2026 cohort reports per-type breakdowns universally but shows no improvement in efficiency reporting, with only 2 of 9 systems reporting any efficiency metric.Per-type reporting rose from 60% before 2026 to 100% in 2026, while efficiency reporting remained statistically indistinguishable.
  • B Memory Substrate Implementation Details: The implementation appendix specifies eleven memory substrates and their write, read, and management procedures.The substrates cover dense and sparse indices, text records, structural and hierarchical stores, refinement memories, weight updates, and activation-compatible context mechanisms.
  • B.1 External Substrates: External substrates range from LLM-free dense and BM25 retrieval to gist filtering, linked notes, hybrid knowledge-graph retrieval, hierarchical summaries, and refined strategies or skills.Dense retrieval uses 1536-dimensional embeddings and cosine similarity; sparse retrieval uses a BM25 inverted index, while several other substrates add auxiliary-LLM processing.
  • B.2 Internal Substrates: Internal substrates include LoRA adapter tuning without explicit retrieval, full-history concatenation with truncation, and episode-clustered re-prefill using a 25% token budget.The EpiCache variant re-prefills selected utterances because the released custom kernels do not support Qwen3’s hybrid-attention architecture.
  • B.1 External Substrates: Auxiliary-LLM costs can be substantial: they are one to two orders above the lightest retrieval baselines, whereas M2 makes zero auxiliary-LLM calls.On a full LoCoMo run, M2 finishes in roughly ten minutes; the cost comparison excludes embedding.

B.3 Memory Substrate and Method Exclusion · C Benchmark Setups and Bank Alignment · C.1 Benchmark Setups and Capability Definitions

The appendix excludes three production-grade memory systems from the controlled ablation because their auxiliary-LLM costs are excessive, while defining benchmark capabilities, setups, and retrieval-bank alignment across user- and agent-centric evaluations. It covers LoCoMo, four MemoryAgentBench capabilities, and embodied-planning and code benchmarks that probe accumulation, recovery, scalability, and staleness.

  • B.3 Memory Substrate and Method Exclusion: MemGPT, Mem0, and Zep were excluded on budget grounds after auxiliary-LLM costs were measured on a single end-to-end LoCoMo run.The run used ten conversations, 1,986 queries, and GPT-4o-mini as the auxiliary model; embedding calls and latency were excluded.
  • B.3 Memory Substrate and Method Exclusion: 2,700 to 9,000 auxiliary LLM calls and 11 to 23 hours of wall-clock time characterized each excluded substrate on one LoCoMo run.These budgets were one to two orders of magnitude above lighter retrieval baselines because writes and reads trigger multiple auxiliary calls.
  • B.3 Memory Substrate and Method Exclusion: The exclusion preserves controlled cross-substrate ablation because lighter Table 4 approximations represent Mem0 distillation, Zep temporal graphs, and MemGPT tiered buffers.M3 approximates Mem0 and MemGPT-related record-store semantics, while M5 overlaps with Zep’s entity/relation indexing.
  • C Benchmark Setups and Bank Alignment: The appendix expands benchmark descriptions with capability definitions, evaluation splits, pool construction, and semantic alignment audits for agent-centric retrieval banks and evaluation pools.This frames the benchmark setup and bank-alignment coverage of the appendix.
  • C.1 Benchmark Setups and Capability Definitions: LoCoMo contains ten multi-session dialogues and 1,986 questions across five categories, providing an aggregate quality indicator that mixes hardness dimensions.Its design combines multiple difficulty dimensions within one benchmark.
  • C.1 Benchmark Setups and Capability Definitions: MemoryAgentBench defines Accurate Retrieval, Long-Range Understanding, Test-Time Learning, and Conflict Resolution to assess retrieval, signal preservation, inspectability, and temporal updating.AR uses 300 LongMemEval-S∗ queries of ∼355K tokens each; CR also serves as the scalability probe by sweeping content length, while parametric substrates lack TTL’s inspectability.
  • C.1 Benchmark Setups and Capability Definitions: ALFWorld provides 134 valid-unseen embodied-planning tasks with binary environment rewards under within-episode accumulation and cross-episode recovery regimes.The shared cross-episode bank contains 100 AgentGym successes and 100 QWEN3-32B-AWQ real failures.
  • C.1 Benchmark Setups and Capability Definitions: BigCodeBench-Hard evaluates 148 code tasks with real PYTEST execution using an offline pool of cross-task solutions and harvested failures, after removing Hard-set overlap.Retrieved entries appear as labelled SUCCESSFUL and FAILED blocks, stressing accumulation and staleness.

C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard · D Metrics Definition and Assessment · D.1 Notation

The ALFWorld and BigCodeBench-Hard banks are semantically aligned with their evaluation pools well above random controls, while avoiding direct instance overlap. The appendix defines unified notation and separates write, read, and management phases for metric assessment.

  • C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard: ALFWorld uses 200 balanced trajectories to support evaluation on the canonical n=134 VALID_UNSEEN split.The bank combines 100 AgentGym successes and 100 QWEN3-32B-AWQ real failures across six task types.
  • C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard: 0.761 top-5 mean cosine shows ALFWorld test goals are close to neighborhoods of bank items.30.6% of items fall in [0.95, 1.00] and 35.8% in [0.85, 0.95).
  • C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard: 0.068 mean cosine for the ALFWorld random-bank control is an order of magnitude below the real bank.The prefix filter excludes the actual VALID_UNSEEN environment instance, so retrieval supports strategy transfer rather than answer copying.
  • C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard: BCB-Hard retrieval uses a cross-task pool for n=148 Hard instructions after removing Hard-set overlap before runs.The pool contains solution snippets from non-Hard tasks and failures from preliminary runs.
  • C.2 Bank–Test Semantic Alignment for ALFWorld and BCB-Hard: 25% of Hard tasks have a bank example with cos ≥0.65, 40% fall in [0.55, 0.65), and 35% fall below 0.55.The groups represent concrete reusable scaffolding, shared libraries or patterns, and genuinely out-of-pool tasks, respectively.
  • D Metrics Definition and Assessment: The appendix defines q, y, predicted answer ˆy, retrieved items R, gold evidence identifiers G, and normalization norm(·) for metric computation.It organizes metrics into Performance and Efficiency families and distinguishes benchmark-specific from substrate-level reporting.
  • D.1 Notation: Evaluation separates write and read phases, while asynchronous maintenance forms a third management phase accounted for under E10 and E11.The read phase is the per-query read→generate loop; management examples include A Mem evolution, MemSkill clustering, and LoRA optimiser steps.

D.2 Performance metrics … E.1 Full Table for the Locomo Benchmark

The paper defines a unified taxonomy of performance and efficiency metrics, aggregation conventions, and comparability rules, then applies them to show distinct accuracy–cost trade-offs among LoCoMo memory substrates. Results favor regime-aware substrate choices rather than a universally dominant memory design.

  • D.2 Performance metrics: P1 Exact Match compares normalized answers exactly, while P2 Token F1 uses whitespace-token precision, recall, multiset overlap, and assigns zero to empty predictions.P1 is brittle to paraphrase and complements broader metrics.
  • D.2 Performance metrics: P3 applies a brevity penalty while ignoring recall, and P4 uses deterministic GPT-4O-MINI judging to report the fraction of CORRECT labels.P4 rubrics vary by benchmark family.
  • D.3 Efficiency metrics: Efficiency metrics cover memory size, inference, write and retrieval latency, write and read tokens, call counts, and maintenance tokens and calls.Embedding-only substrates have E5= 0, while retrieval latency is undefined for substrates without retrieval.
  • D.3 Efficiency metrics: Retrieved tokens are informational because larger E7 can improve recall while inflating downstream E8, whereas E8–E11 separately count read and maintenance costs.E8 excludes embedding tokens, and substrates without maintenance loops have E10= 0.
  • D.4 Aggregation and reporting conventions: Metrics are averaged per query, episode, problem, or event as appropriate, with LoCoMo question-type and MemoryAgentBench capability breakdowns reported separately.Latency tables print means, while p50 and p90 remain in per-substrate result files.
  • D.4 Aggregation and reporting conventions: Undefined cells remain n/a rather than zero, and reported gaps of ≥0.05 on P4 or task success are unlikely to reverse under resampling.The stated sample sizes are n=1986 for LoCoMo, n=300 for LME-S, n=134 for ALFWorld, and n=148 for BCB-Hard.
  • E.1 Full Table for the Locomo Benchmark: On LoCoMo, M2 reaches 0.33 s/query, M1 reaches 0.77 with a +0.07 P4 dense-encoder advantage, M5 leads P4 at 0.648, and M6 leads Recall@k at 0.797.M5 incurs E12=51M and E15=28 s/query, while M6 uses E5=382K and zero read-phase LLM.

E.2 Full Tables for the LongMemEval-S and MAB Benchmarks · E.3 Full Table for AlfWorld · F Additional Related Work

The appendix tables show that substrate performance varies by benchmark, backbone, storage, latency, and token profile rather than following one universal ordering. Related work further shows that memory systems span diverse substrate families and benchmarks, while existing evaluations often omit comprehensive efficiency accounting.

  • E.2 Full Tables for the LongMemEval-S and MAB Benchmarks: On LongMemEval-S and MAB, substrate rankings vary by task: broad retrieval helps factual QA, whereas excessive retrieval can harm sequential decision-making.The evaluated MAB sub-benchmarks are LRU, TTL, and CR, alongside LongMemEval-S.
  • E.2 Full Tables for the LongMemEval-S and MAB Benchmarks: M4 reaches M4 CR P12=0.080 despite 6.2× store compression, while M5 incurs E5=10.3M and E2≈8.5 s/query on QWEN3-8B.M4’s evolved notes fail to preserve the temporal serial-number cue, and M5 only matches M1 on LME-S SubEM (P12).
  • E.2 Full Tables for the LongMemEval-S and MAB Benchmarks: Storage tiers are nearly model-independent: Activation stores are ≤80 KB, Flat/Text/M4 stores are 1.6–9.7 MB, and heavy stores are ∼22–32 MB.M9’s adapter additionally occupies 80–320 MB.
  • E.3 Full Table for AlfWorld: On ALFWorld, M1 trails NoMem on QWEN3-8B (5.2 vs. 5.7 TSR) but exceeds it on QWEN3-32B-AWQ (27.6 vs. 22.4) and GEMMA-4 (10.4 vs. 7.5).M2 remains close to NoMem at no additional cost, while Text and Structural families provide modest gains on QWEN3-32B-AWQ.
  • E.3 Full Table for AlfWorld: M9 reaches 29.9 vs. 22.4 TSR on QWEN3-32B-AWQ but falls to 3.0 TSR on QWEN3-8B, whereas M11 reaches 11.9 vs. 5.7.M9 is excluded on GEMMA-4 because it is incompatible with A4B MoE routing; M11 exceeds NoMem on every backbone.
  • F Additional Related Work: Related work covers flat retrieval, temporal and graph stores, hierarchies, refinement-based memories, KV-cache activation management, and adapter-based parametric updates.These systems improve recall, inspectability, abstraction, or reusable skills but introduce choices such as summarization, graph construction, entity resolution, hierarchy depth, and traversal.
  • F Additional Related Work: Existing benchmarks span user-centric and agent-centric demands, but evaluations often omit write, management, storage, and comprehensive token-cost accounting.Mem0 reports token consumption and p95 latency without complete storage or write overhead, while Zep reports latency and token cost without full construction-cost accounting.

G Limitations · H Declaration of LLM Usage

The evaluation excludes three production-grade memory systems because their auxiliary-LLM costs would overwhelm the cross-substrate comparison, and it modifies two reference implementations to avoid confounds and unsupported operations. The authors report using LLMs only for routine writing assistance, while the paper includes BigCodeBench-Hard latency, storage, and token-accounting tables.

  • G Limitations: MemGPT, Mem0, and Zep are excluded because their 2,700–9,000 auxiliary-LLM calls and 11–23-hour LoCoMo runtimes would dominate comparison costs.Their retrieval ideas are represented in the main comparison by lighter substrates.
  • G Limitations: M8 replaces MemSkill’s PPO-trained controller with a zero-shot LLM consuming all refined skill bundles prefilled into context.This avoids introducing RL training into an otherwise non-RL cross-substrate comparison.
  • G Limitations: M11 replaces Epi-Cache’s in-place KV-cache edits with utterance-level re-prefill because Qwen3’s hybrid attention architecture does not support the reference operation.The supplied passage ends mid-sentence, so the implementation rationale is reported only to the extent stated.
  • H Declaration of LLM Usage: The authors state that LLMs were used only for grammar polishing, phrasing refinement, and LaTeX formatting.They were not used to generate research ideas, design experiments, analyze results, or produce empirical findings.
  • H Declaration of LLM Usage: The paper attributes the harness design, substrate implementations, metric taxonomy, and analysis to the authors’ original scientific contributions.This statement accompanies the declaration that LLMs did not produce the paper’s empirical findings.
  • H Declaration of LLM Usage: Table 17 reports BigCodeBench-Hard latency and storage across substrates and models.The supplied passage provides the table title but no numerical entries.
  • H Declaration of LLM Usage: Table 18 reports BigCodeBench-Hard token accounting across substrates and models.The supplied passage provides the table title but no numerical entries.
Loading 2608.15008v1…