PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Tax: Dissecting the Net Value of Agent Memory

Xuan He, Sining Zhoubian, Ruozhen Yang, Yuhao Wen, Zhineng Wang, Yuyang Hu, Ke Yang, Yuyu Zhang, Ping Nie, Jiawei Han, ChengXiang Zhai

arXiv:2610.34073v1cs.CL

TL;DR

The net value of building memory over direct retrieval from raw history remains unclear. The paper evaluates memory systems against naive RAG and dissects gains and losses as memory benefit and tax. Many systems underperform RAG in the evaluated settings, motivating attention to memory’s net value and its evaluation.

  • Problem

    The value of increasingly diverse memory operations over direct retrieval from raw history remains unclear, and prior comparisons often overlook naive RAG.

    Source: §1 · p. 1

  • Method

    The paper evaluates memory against naive RAG, separates benefit from tax, analyzes tax from system and evaluation perspectives, and extends the view to other memory settings.

    Source: §1 · pp. 1–2

  • Results

    Across three benchmarks with retrieval-friendly and more complex queries, many memory methods underperform naive RAG, indicating negative net value in these settings.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The paper argues for assessing memory by its net value and its benefit-tax balance to inform more reliable memory design and evaluation.

    Source: §8 · p. 8

    The benchmarks are primarily static, offline, and English-language, leaving long-lived deployments with interleaved queries and multilingual interactions underexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/

1 Introduction

The paper asks whether long-term memory adds value over direct retrieval from raw history, finding many systems underperform naive RAG in diverse settings. It frames memory’s gains and losses as benefit and tax, then examines their sources and broader applicability.

1 Introduction

  • Prior work often compares memory designs with one another while overlooking naive RAG, leaving memory’s marginal value over direct retrieval underexamined.The paper identifies this gap as motivating its central question.Source: §1 · p. 1
  • Many memory systems underperform naive RAG across diverse settings, indicating negative net value when memory tax outweighs benefit.Memory can also answer queries RAG misses, while losing queries RAG solves.Source: §1 · p. 1
  • Memory tax and benefit capture opposing effects of deviations from raw history: organization can help, while lost or obscured evidence can harm performance.Their balance defines memory’s net value, reflected in external performance.Source: §1 · pp. 1–2
  • The paper examines memory tax through faithfulness, accessibility, and robustness, and notes that retrieval-friendly benchmarks with high RAG scores leave limited headroom for memory benefits.Faithfulness concerns information survival, accessibility concerns recall of preserved information, and robustness concerns reliability across backbone changes.Source: §1 · p. 2
  • The benefit-tax view is extended to experiential and world-model memory, with design implications focused on changes usable in existing memories without new architectures.The paper presents the view as generalizable beyond particular memory forms.Source: §1 · p. 2

2 Preliminary

The preliminary formulation treats memory as a transformation of interaction history followed by query-conditioned retrieval and answer generation; naive chunk-based RAG serves as the reference.

2 Preliminary

  • A memory system transforms interaction history H into persistent memory M through an ingestion process I, then retrieves relevant units C for a query q.The answer is generated from the query and retrieved units.Source: §2 · p. 2
  • Memory may take forms such as summaries, extracted facts, structured records, or episodic entries.Source: §2 · p. 2
  • Naive chunk-based RAG minimally processes raw history and retrieves relevant chunks directly, providing the reference for measuring memory benefit and tax.Source: §2 · p. 2

3 Experiments

The experiments compare representative memory systems with raw-history RAG across benchmarks and model backbones, aligning retrieval budgets for fairness. Many systems fail to beat RAG in many cases.

3.1 Setup

  • The benchmarks are LongMemEval-s, PersonaMem-v2, and MINT-Eval, spanning direct recall and more complex aggregation or multi-interference queries.Cost constraints led to stratified evaluation subsets of 20% of PersonaMem-v2 and 10% of MINT-Eval.Source: §3.1 · p. 2
  • The study includes flat-store, hierarchical/lifecycle, graph-based, and trained-policy memory systems, with raw-corpus RAG as the main memory-free baseline.It also includes single- and two-round RAG and vanilla references such as No Context and Gold Context.Source: §3.1 · p. 2
  • Figure 1 plots accuracy against token cost, with bubble size representing memory-store tokens.The experiments also report wall-clock time, LLM calls, and storage size alongside accuracy.Source: Figure 1 · p. 3, §3.1 · p. 3
  • Retrieval budgets are aligned using Retrieval Scope, the fraction of a system’s retrievable-store tokens exposed to the LLM, tuned to be roughly no lower than one-round RAG.Systems with only a few memory units retain a fixed top-k.Source: §3.1 · p. 3

3.2 Results

  • Many memory systems fail to beat RAG in many cases across the three-benchmark evaluation.Table 1 reports judge accuracy, while Figure 1 plots the accuracy–token-cost trade-off; full results are in appendices E.1–E.3.Source: §3.2 · p. 3, Table 1 · p. 3

4 Memory Tax

Relative to naive RAG, memory can help on some queries but lose others, and many evaluated memory systems underperform RAG overall. The section traces this tax to information loss, reduced accessibility, backbone sensitivity, and limited benchmark headroom.

4.1 Overview

  • Many memory systems underperform naive RAG overall: memory solves some queries RAG misses but loses others RAG solves, leaving the tax greater than the benefit in these cases.Memory benefit and tax are defined relative to naive RAG; the Venn plot compares query-level corrections.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2 Memory Tax - Faithfulness: How Well Is the Information Preserved?

  • Information loss during memory construction directly contributes to memory tax; replaying QA with gold memory units isolates construction losses, while remaining gaps indicate access problems can persist beyond construction.The analysis compares gold raw turns with corresponding gold memory units and also annotates information loss at the fact level.Source: §4.2 · p. 4, Figure 3 · p. 4

4.3 Memory Tax - Accessibility: How Easily Can Stored Information Be Accessed?

  • Memory systems recover fewer queries than RAG in the first retrieval round, and fragmentation, dilution, or rephrasing can make preserved evidence harder to access.The analysis tracks rounds needed to retrieve all gold memory units and compares query alignment for gold raw turns versus gold memory units.Source: §4.3 · p. 5, Figure 4 · p. 5, Figure 5 · p. 5

4.4 Memory Tax - Robustness: How Robust Is the Memory Across Backbones?

  • With construction fixed, changing the QA backbone causes larger performance variation than naive RAG for many systems, while changing the construction backbone also varies scores.The authors attribute potential added sensitivity to model-dependent memory operations that introduce variability relative to raw history.Source: §4.4 · p. 5, Figure 6 · p. 6

4.5 Benchmark Headroom Further Exposes the Memory Tax

  • Simple retrieval baselines perform well on some benchmarks, including IDF-keyword above 50% and RAG-Hybrid about 70%, leaving limited headroom for memory gains.When retrieval already solves many queries, modest losses from memory processing become more visible; lightweight or no-construction systems can remain competitive, and evaluations could add harder settings.Source: §4.5 · pp. 5–6

5 Generalizing Memory Tax

The benefit-tax view extends beyond personal-agent memory to experiential and world-model memory. Results vary by task domain and memory representation, showing that the value of memory operations depends on the setting.

5.1 Experiential Memory

  • In web tasks, distilling traces into workflows or strategies can outperform naive RAG, while several tool-use memory methods fall behind it.The reported effect varies across task domains, and the magnitude of memory tax depends strongly on task settings.Source: §5.1 · p. 6, Figure 8 · p. 7

5.2 World-Model Memory

  • Directly retrieving and reinjecting past observations achieves strong world-model consistency, while latent- or cache-based memories do not consistently outperform it.The authors suggest additional memory operations may introduce losses that offset their intended benefits.Source: §5.2 · p. 6, Table 3 · p. 7

5 Generalizing Memory Tax

  • The benefit-tax view generalizes to nontrivial memory operations that introduce both beneficial and harmful deviations from source information.The section examines this view in experiential and world-model memory.Source: §5 · p. 6, §5.2 · p. 7

5.2 World-Model Memory

  • The authors leave extensions to other axes, including omni-modality and parameters, for future work.They expect the benefit-tax view to extend beyond the settings examined here.Source: §5.2 · p. 7

6 Design Implications

The section proposes ways to reduce or compensate for memory tax without adding a new architecture. It emphasizes recoverable deviations, retrieval feedback, and anticipating future information demand.

6 Design Implications

  • The proposed implications target changes applicable to existing memory systems and exclude test-time scaling techniques such as expanding retrieval depth or width.The section focuses on reducing or compensating for memory tax without introducing another architecture.Source: §6 · p. 7

6.1 (Memory Construction) Make memory deviations more recoverable.

  • Retrieving from source history alongside normal memory and jointly reranking can restore lost or inaccessible evidence, with Table 4 reporting consistent improvements across several memory systems.The proof of concept adds a lightweight recovery path when memory is insufficient.Source: §6 · p. 7, Table 4 · p. 8

6.2 (Memory Access) Recursive or selfreferential memory for retrieval.

  • Adding past queries and retrieved evidence to memory makes them available for future retrieval, and Table 4 validates this approach.Deeper-round evidence can then be accessed directly for similar information needs, creating feedback between memory access and construction.Source: §6 · p. 7, Table 4 · p. 8

6.3 (Future Direction and Beyond Memory Operations) Try modeling future demand beyond past information.

  • Modeling future demand from past interests, habits, and access patterns may guide selective memory construction between costly advance processing and latency-increasing just-in-time operations.For LongMemEval-s, which has one query per corpus, inter-query mechanisms do not apply.Source: §6 · pp. 7–8

7 Related Works

Prior work spans varied memory representations and management mechanisms, and empirical studies characterize design and performance. This paper instead focuses on the net value introduced by memory processing.

Long-Term Agent Memory

  • Long-term memory systems use flat, hierarchical, and graph-based representations, including short-/long-term layers and filesystem-tree designs [Kang et al., 2025; Fang et al., 2026a; Jiang et al., 2026a; Yang et al., 2026a].The passage also cites examples of flat stores [Liu et al., 2026b; Yang et al., 2026b; Meng et al., 2026] and filesystem-tree designs.Source: §7 · p. 8
  • Memory operations may follow predefined harnesses, dynamic LLM-agent decisions, or learned policies [Wang and Chen, 2025; Xu et al., 2026a; Yan et al., 2026c; Yu et al., 2026a; Yan et al., 2026b].The work studies these systems through the net value of gains and losses introduced by memory operations.Source: §7 · p. 8

Empirical Studies of Agent Memory

  • Prior empirical studies examine memory design, deployment, performance-cost trade-offs, and resource use, whereas this work analyzes losses relative to direct retrieval and derives system-agnostic design implications.The paper introduces memory tax to characterize those losses and considers both system and evaluation settings.Source: §7 · p. 8

8 Conclusion

The paper evaluates memory against direct retrieval from raw history and finds many methods underperform naive RAG in the studied settings. It frames memory's gains and losses as benefit and tax, analyzes the tax, and extends the view to other memory scenarios.

8 Conclusion

  • Across three benchmarks with retrieval-friendly and complex queries, many memory methods underperform naive RAG, indicating negative net value in these settings.The paper dissects memory's net value into benefit and tax, analyzes tax from system and evaluation perspectives, and generalizes the view to experiential and world-model memory.Source: §8 · p. 8

Limitations

The evaluation has several scope limits: its baselines are representative rather than exhaustive, its benchmarks are mostly static, offline, and English-language, and resource constraints limit models and samples.

Limitations

  • The baseline selection is representative rather than exhaustive, and system-specific tuning could improve performance.The authors also note that they follow official configurations wherever possible.Source: Limitations · p. 9
  • The benchmarks are primarily static, offline, and English-language, leaving interleaved queries and multilingual interactions in long-lived deployments underexplored.Omnimodal, latent, and parametric memory are also left for future work.Source: Limitations · p. 9
  • Computational and API costs limit the evaluation: it omits stronger frontier models beyond GPT-5.6 Luna (low) and uses stratified benchmark subsets for that model.The subsets are 20% of PersonaMem-v2, 10% of MINT-Eval, and 20% of LongMemEval-s questions.Source: Limitations · p. 9

A.1 Benchmark Details

The appendix documents the evaluation benchmarks, baselines, scoring and cost measures, and implementation choices; it also explains a model-specific scoring pattern and the validation of the RAG reference.

A.1 Benchmark Details

  • LongMemEval-s uses chat histories with about 50 sessions per question, PersonaMem-v2 tests implicit persona and preferences, and MINT-Eval tests memory under multi-target interference.The evaluation uses the full LongMemEval-s set except a stratified 100-corpus subset for GPT-5.6 Luna, and samples 20% of PersonaMem-v2 and 10% of MINT-Eval.Source: §A.1 · p. 15

A.2 Baseline Details

  • The baselines include no-context and full-context references, gold context where available, chunk-based naive RAG, and memory systems grouped by storage or management approach.The gold-context baseline is omitted on MINT-Eval because verified gold evidence is available for only about half of its questions.Source: §A.2 · p. 15

A.3 Evaluation Metrics

  • Answers are scored with an LLM-as-Judge, while reported system costs include tokens, normalized wall-clock time, LLM calls, and memory-store size.PersonaMem-v2 is scored by matching the option letter to human labels.Source: §A.3 · pp. 15–16

A.4 Implementation

  • Retrieval Scope measures the fraction of tokens in retrievable memory units exposed to the LLM, and top-k is adjusted so systems are no lower than RAG (1-round).The method counts additional retrieval-time LLM exposure; systems with very few large units are an exception and use a fixed top-k.Source: §A.4 · p. 16

A.5 More Explanations

  • GPT-5.6 Luna usually scores lower than Qwen3.6-27B on PersonaMem-v2, which the authors attribute to its preference for generic options over personalized ones in that benchmark.They note that PersonaMem-v2 often rewards explicit user preferences, unlike the other two benchmarks.Source: §A.5 · p. 16

B Configuration Validation

  • Naive RAG remains strong across component ablations, while many memory systems underperform it; the authors validate the reference with ablations, prior-work review, and memory-configuration checks.Most RAG variants stay within a few points of the default, and some memory systems remain below even the more challenging variants.Source: B Configuration Validation · p. 17

B.1 Ablations on RAG Baseline

The appendix tests the stability of the RAG reference and validates comparison settings, then analyzes memory tax through evidence preservation, retrieval accessibility, and robustness.

B.1 Ablations on RAG Baseline

  • Most naive RAG ablations stay within a few points of the default, and even the largest drops remain competitive with several memory systems.The largest drops occur with fixed-size windows that split turns or with BM25-only retrieval.Source: §B.1 · p. 17, Table 7 · p. 17

B.2 RAG Implementation in Prior Works

  • Only three reviewed memory papers report RAG results, and their retrieval setups vary; the authors argue that a standard hybrid retrieval-and-reranking pipeline makes RAG a strong reference.They conclude that prior studies may have underestimated naive RAG and should adequately evaluate retrieval controls.Source: §B.2 · pp. 17–18

B.3 Memory Baseline Config Offset Validation

  • Aligning retrieval scope generally improves memory-baseline performance, while cases requiring a reduced top-k show only minor degradation.Most adjustments increase top-k beyond repository defaults to account for differences in memory-unit granularity.Source: §B.3 · p. 18

Oracle raw context vs. oracle memory units

  • Oracle comparisons separate loss during memory construction from retrieval failure; several systems show raw-context versus oracle-memory gaps indicating information loss.For MAGMA and LIGHTMEM, oracle-memory performance is close to oracle-raw performance but much higher than normal retrieval, suggesting preserved evidence is harder to recover.Source: Oracle raw context vs. oracle memory… · p. 19

C.1 Overview

C.3 Memory Tax - Accessibility

  • Memory systems generally recover fewer gold memory units than RAG in the first retrieval round, even when the evidence is preserved in memory.The analysis replays retrieval rounds until all gold units are recovered or the round limit is reached.Source: §C.3 · p. 22

C.4 Memory Tax - Robustness

  • The robustness analysis varies the memory-construction and QA backbones and compares score drops against RAG under the same QA-model downgrade.For retrieval methods without LLM involvement, the retrieved units are held fixed across QA backbones.Source: §C.4 · p. 22, Figure 15 · p. 22

C.5 Benchmark Headroom Further Exposes the Memory Tax

  • The appendix evaluates retrieval-only controls across a broader set of long-term memory benchmarks, including multimodal benchmarks, using captioned histories for image inputs.Simple retrieval baselines and their retrieval scopes are specified, but these passages do not report their outcomes.Source: §C.5 · p. 23

C.6.1 Experiential Memory

The appendix extends the benefit-tax analysis to experiential and world-model memory, and uses cases to illustrate how memory operations can help or hinder individual queries.

C.6.1 Experiential Memory

  • The listed evaluations include ALFWorld household tasks (300 / 134), WebShop simulated product search and buying (1,000 / 500), Mind2Web website navigation (1,009 / 1,341), and BigCodeBench Python programming (992 / 148).Source: §C.6.1 · pp. 23–24

C.6.2 World-Model Memory

  • World-model memory is evaluated on revisit consistency and object-state recall, with baselines ranging from past-frame retrieval to compressed or learned memory representations.The appendix reports long-horizon consistency results for Context-as-Memory Test and MemoBench.Source: §C.6.2 · p. 25, Table 18 · p. 25

D.1 Memory Benefit

  • Memory can answer queries that RAG misses through focused fact extraction, cross-session linking, explicit preference exposure, temporal anchoring, forgetting, or processing heterogeneous inputs.These are representative benefit cases, despite memory underperforming RAG overall in many cases.Source: §D.1 · p. 26

D.2 Memory Tax - Faithfulness

  • Faithfulness case studies span no loss, partial loss, major loss, and cases where no corresponding memory unit remains.The examples come from the annotation analysis.Source: §D.2 · p. 26

D.3 Memory Tax - Accessibility

  • Accessibility examples show preserved evidence arriving later because turns are fragmented, diluted by unrelated content, or rephrased away from query wording.The case studies contrast retrieval rounds for memory evidence with raw-turn retrieval.Source: §D.3 · p. 26

D.4 Direct Examples for the Double-edged Memory Mechanisms

  • The same memory mechanism can help on one question and hurt on another, as paired cases contrast memory wins and losses against RAG.The paired examples hold the mechanism constant while changing the query outcome.Source: §D.4 · p. 26, Figure 25 · p. 31, Figure 26 · p. 32

E.1 Full Results of LongMemEval-s

The appendix reports aggregate accuracy and cost metrics alongside Pareto plots for three benchmark–model settings, then provides QA and LLM-as-judge prompt variants.

E.1 Full Results of LongMemEval-s

  • LongMemEval-s results report benchmark-level judge accuracy and cost metrics, with Pareto plots comparing accuracy against token cost and LLM-call count.The plots encode memory-store size through bubble size; results use 100 of 500 corpora for GPT-5.6 Luna (low).Source: Table 19 · p. 33, Figure 27 · p. 33

E.2 Full Results of PersonaMem-v2

  • PersonaMem-v2 results report benchmark-level judge accuracy and cost metrics, with Pareto plots comparing accuracy against token cost and LLM-call count.The plots encode memory-store size through bubble size; results use 40 of 200 corpora for Qwen3.6-27B.Source: Table 20 · p. 33, Figure 28 · p. 34

E.3 Full Results of MINT-Eval

  • MINT-Eval results report benchmark-level judge accuracy and cost metrics, with Pareto plots comparing accuracy against token cost and LLM-call count.The plots encode memory-store size through bubble size; results use 60 of 595 corpora for Qwen3.5-4B.Source: Table 21 · p. 34, Figure 29 · p. 34

F.1 QA Prompts

  • The QA prompts cover general open-ended, preference, and multiple-choice questions, with profile sections omitted when unavailable.The general prompt directs concise answers, date and duration formatting, recency-based conflict handling, combining scattered evidence, and abstention when information is absent; preference questions drop the unavailable-information rule.Source: Table 22 · p. 35, Table 23 · p. 35, §F.1 · p. 35

F.2 LLM-as-Judge Prompts

Loading 2610.34073v1…