PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryTax: The Bitter Lesson of Agent Memory

Zedong Peng

arXiv:2610.29823v1cs.CLcs.AI

TL;DR

The paper asks whether designed agent memory still helps once models can search conversation histories themselves. It compares memory methods and coding agents on one LoCoMo conversation, finding that memory helps only the weakest model while Pi with Claude Opus 5.5 answers every sound-key question for less than full context costs.

  • Problem

    The paper asks whether designed memory still helps once a model is strong enough to search the raw conversation history itself.

    Source: §1 · p. 1

  • Method

    The authors compare four memory methods and three coding agents across GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5 on one LoCoMo conversation, recording rollout costs.

    Source: §1 · p. 1

  • Results

    On this LoCoMo conversation, Mem0 helps GPT-4o-mini but not the two stronger models, while Pi with Claude Opus 5.5 answers every sound-key question for less than full context costs.

    Source: §6 · pp. 4–5

  • Takeaways & Limitations

    The results suggest that strong models may need less designed memory and better tools for searching the raw record, though designed memory may remain a cost trade-off when many questions share one history.

    Source: §6 · p. 5

    The findings come from one conversation and one run per cell, use a GPT-4o-mini judge, and include comparisons affected by unequal access to photo captions.

    Source: §6 · p. 5

Abstract

from arXiv · show

Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives; coding agents keep the raw history and let the model search it with its own tools. We ask whether designed memory still helps once the model is strong enough to search for itself. On one LoCoMo conversation (419 turns, 152 questions), we run four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, and record what every rollout costs. This is a short empirical study, with one conversation and one run per cell. Agent memory is a costly patch for a weak model: Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than a whole full-context rollout. Agent memory also has a ceiling, set by the model that writes it: a stronger reader does not recover what a weaker writer left out. With Claude Opus 5.5, Pi tops the cost-accuracy frontier, scoring 99% for less than half the cost of full context; its one miss is a wrong answer key. Part of that lead comes from photo captions that only the agents could read: on the 88 questions whose evidence has no photo, Pi and full context differ by one answer.

1 Introduction

The paper asks whether designed agent memory still helps once models can search raw conversation histories themselves. It compares memory systems and coding agents on a LoCoMo conversation and finds that memory’s benefits depend on model strength, while Pi reaches the cost–accuracy frontier with Claude Opus 5.5.

1 Introduction

  • Designed memory selects what to retain before questions arrive, whereas coding agents keep raw history and search it with their own tools [1] [2].Source: §1 · p. 1
  • The study compares full context, BM25, Mem0 and Zep with three coding agents and three models on one LoCoMo conversation.It records the cost of each rollout.Source: §1 · p. 1
  • Memory helps GPT-4o-mini, but its benefit disappears with stronger models, and Pi with Claude Opus 5.5 reaches the cost–accuracy frontier.Pi answers every question with a sound answer key for less than half the cost of full context; its remaining lead partly reflects photo captions unavailable to memory methods.Source: §1 · p. 1
  • The study is limited to one conversation, one run per cell and a GPT-4o-mini judge.Within those limits, the authors suggest strong models need less designed memory and better tools for searching raw records.Source: §1 · p. 1
  • Figure 1 plots correct rate against cost per rollout, with each badge representing a method-and-model run.The stepped line marks the highest correct rate reached at or below each cost; cost uses a log axis.Source: Figure 1 · p. 2

2 Setup

The evaluation uses one LoCoMo conversation with 419 turns and 152 questions, comparing memory methods and coding agents across three models. It records whole-rollout costs, including memory builds and agent calls.

2 Setup

  • The study evaluates 21 cells on LoCoMo conv-26, a 419-turn, 19-session conversation with 152 questions.Questions cover single-hop, multi-hop, temporal and open-domain tasks; adversarial questions are excluded, and the history fits every model’s context window.Source: §2 · p. 2
  • Each cell uses one model to write and answer from a Mem0 or Zep store, or to drive a coding agent.The models are GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5.Source: §2 · p. 2
  • The memory methods receive turn text only, while coding agents receive the official conversation file with photo captions included.116 turns share a photo, and 64 questions have evidence in such a turn.Source: §2 · p. 3
  • A rollout includes the build and all 152 questions, with cost counting model-call list prices but excluding the judge.Each cell runs once; tokens include cached input, while embedding tokens are not counted.Source: §2 · p. 3
  • For each question, the memory methods answer once, while an agent makes one call per step of its loop.GPT-4o-mini grades answers using Mem0’s 2025 judge rubric [1]; empty answers count as wrong.Source: §2 · p. 3

3 Agent memory is a costly patch for a weak model

Mem0 beats full context only with GPT-4o-mini; with stronger models, designed memory stops helping. Its build costs more than a full-context rollout, and Mem0 and Zep do not reach the cost–accuracy frontier.

3 Agent memory is a costly patch for a weak model

  • With GPT-4o-mini, Mem0 scores 78%, versus 74% for both full context and BM25; Zep scores 43%.The agents score 63% (Pi), 28% (Codex) and 29% (Claude Code).Source: §3 · p. 3
  • With Claude Opus 5.5, Pi scores 99%, Codex 98% and Claude Code 97%, while Mem0 and full context each score 95%.With GPT-6 Luna, full context scores 92%, BM25 82%, Mem0 80% and Zep 51%.Source: §3 · p. 3
  • Mem0’s build costs $0.56 versus $0.43 for full context with GPT-4o-mini, $0.45 versus $0.29 with GPT-6 Luna, and $25.88 versus $17.84 with Claude Opus 5.5.Its questions are cheaper, but the build breaks even only after 372, 545 and 596 questions, respectively; LoCoMo asks 152.Source: §3 · p. 3

4 Agent memory has a ceiling

The model that writes a memory store limits what later readers can answer. Across both Zep and Mem0, stronger readers do not fully recover information omitted by weaker writers.

4 Agent memory has a ceiling

  • The experiment swaps readers across stores because retrieval itself calls no language model.Mem0 and Zep stores are written by the same model that later reads them in the main comparison.Source: §4 · p. 3
  • On Zep’s GPT-4o-mini graph, GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5 score 43%, 43% and 45%; on Claude Opus 5.5’s graph, they score 66%, 71% and 87%.For Mem0, the corresponding scores are 78%, 83% and 84% from GPT-4o-mini’s store, and 89%, 91% and 95% from Claude Opus 5.5’s store.Source: §4 · p. 3

5 With Claude Opus 5.5, Pi tops the frontier

With Claude Opus 5.5, Pi leads the cost–accuracy frontier, scoring 99% for $6.95 per rollout versus 95% for $17.84 with full context. Most of Pi’s advantage over full context comes from photo captions; without photo evidence, their scores differ by one answer.

5 With Claude Opus 5.5, Pi tops the frontier

  • 99% for $6.95 per rollout puts Pi with Claude Opus 5.5 above every other cell; full context scores 95% for $17.84.Pi’s one miss is a wrong answer key: the conversation says Melanie shared the painting, not Caroline.Source: §5 · pp. 3–4
  • Figure 2 accumulates rollout cost and correct answers from the build through questions ordered by cost; strong lines mark cells unbeaten on both measures.Badges show the unsmoothed Table 2 totals, while curves use a nine-point moving average.Source: Figure 2 · p. 4
  • Pi reads about 10,000 input tokens per question, 7,900 cached, versus 28,900 for full context.Codex and Claude Code score 98% and 97% for $25.41 and $28.41, respectively.Source: §5 · p. 4
  • On the 88 questions without photo evidence, Pi scores 100% and full context, Mem0, Codex, and Claude Code each score 99%.Six of Pi’s seven answers beyond full context rely on turns with photos whose captions Pi could read and full context could not.Source: §5 · p. 4

6 Discussion and limitations

The study suggests that designed memory is costly for strong models and that the store writer limits what readers can answer. Its conclusions are bounded by a small evaluation, while the discussion points toward model-driven search over raw records.

6 Discussion and limitations

  • On this LoCoMo conversation, Mem0 helps GPT-4o-mini but not GPT-6 Luna or Claude Opus 5.5, and a stronger reader does not recover what a weaker writer left out.With Claude Opus 5.5, Pi answers every question whose answer key is sound for less than full context costs.Source: §6 · pp. 4–5
  • Figure 3 uses colors for stores and arrows for cross-reader tests, distinguishing each store’s writer reading it from other models reading it.The filled badge marks the writer reading its own store.Source: Figure 3 · p. 5
  • The study uses one conversation and one run per cell; differences of a few questions may not hold in a second run.Other boundaries include a GPT-4o-mini judge, unequal access to photo captions, histories that fit every model’s context window, and default memory settings.Source: §6 · p. 5
  • The discussion points toward exact, semantic, and vector search over the raw record, driven by the model itself.Automated memory or harness search still fixes what is found before the question arrives; designed memory may remain a cost trade-off when many questions share one history.Source: §6 · p. 5

A Full results

The results visuals organize cost and correctness by method and model, compare readers using different writers’ stores, and split accuracy by photo evidence. Figure 4 also plots rollout cost against turns and tokens.

A Full results

  • Table 2 organizes each method-model cell by build cost, average per-question cost, and correct-answer rates across LoCoMo categories.It also records reasoning effort, with no effort setting for GPT-4o-mini.Source: Table 2 · p. 7
  • Table 3 compares correct rates when readers answer from stores written by different models, with same-model writer-reader cells bolded.Source: Table 3 · p. 7
  • Figure 4 plots Figure 2’s cells against LLM calls and tokens on linear axes, marking cells no other cell beats on both cost and correct answers.Questions are ordered by the plotted cost measure.Source: Figure 4 · p. 8
  • Table 4 separates correct rates for the 64 questions with photo evidence from the other 88 questions.Source: Table 4 · p. 9

B Protocols

Each agent question runs as a fresh Harbor task in a fresh container containing only the conversation file. Agent runs use default tools and model-specific execution settings, with retries for trials that fail or produce no answer.

B Protocols

  • Memory-method requests use high reasoning effort for GPT-6 Luna and Claude Opus 5.5.Mem0 and Zep use their default extraction and retrieval settings.Source: B Protocols · p. 10
  • Each agent question is a Harbor 0.24.0 task in a fresh container holding the conversation file without questions or answers.The agent’s last message is its answer; the container can reach only the model endpoint, and unsuccessful trials are rerun.Source: B Protocols · p. 10
  • GPT-6 Luna and Claude Opus 5.5 run at high reasoning effort, while GPT-4o-mini has no effort setting.Codex uses code mode with GPT-6 Luna; web search is off for Codex and Claude Code, and unavailable to Pi.Source: B Protocols · p. 10
  • Agent questions may take 20 minutes, and Pi and Claude Code are limited to at most 100 turns.Source: B Protocols · p. 10

C Prompts

The prompts instruct agents to answer from the conversation briefly and guide a separate judge to label generated answers against gold answers. The judge is told to accept answers that match the gold answer’s topic or time period despite differences in length or format.

C Prompts

  • The agent prompt asks for the shortest possible answer, using conversation wording where possible and omitting subjects.Timing questions should use the conversation date as reference.Source: C Prompts · p. 10
  • The judge labels generated answers CORRECT or WRONG against a gold answer describing something one user should know about the other.Its output is constrained to a JSON label.Source: C Prompts · pp. 10–11
  • The judge accepts a longer generated answer when it touches the same topic as the concise gold answer.The prompt illustrates this with a shell-necklace answer.Source: C Prompts · p. 11
  • For time questions, the judge accepts different date formats or relative references when they indicate the same date or period as the gold answer.Source: C Prompts · p. 11
Loading 2610.29823v1…