When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

Shweta Mishra; Shashank Mishra

Source-linked AI summary

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

Shweta Mishra, Shashank Mishra

arXiv:2609.05441v1Jul 26, 2026cs.AI

TL;DR

Existing memory evaluations mainly test conversational recall rather than whether remembered facts change tool-using agents’ actions and at what cost. MERIT addresses this gap with leak-checked episodic tasks, controlled corruption, a difficulty ladder, and explicit memory-cost accounting. Across 23,440 scored episodes, memory substantially improves dependent-task success, while updated-fact performance and economic efficiency vary sharply by memory architecture and implementation.

Problem
Prior benchmarks measure conversational recall, leaving limited evidence about whether remembered facts improve tool-executing agents and whether memory’s marginal utility justifies its cost.
Method
MERIT evaluates episodic tool-use tasks across three domains with leak-verified dependencies, updated-fact difficulty, controlled corruption, shared agent scaffolds, and full token-and-dollar metering.
Results
Memory architectures separate sharply on task execution: update-on-write memories remain robust on revised facts, while retrieval is unstable and agents use correctly retrieved facts only inconsistently.
Takeaways & Limitations
Agent-memory evaluation should measure whether memory changes actions, how stale or correct facts are handled, and the utility gained per cost rather than conversational recall alone.
Takeaways & Limitations
Findings are limited to three synthetic domains with scripted users, 10 arcs per cell, and one representative implementation per memory family.

Abstract

from arXiv · show

Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dialogue history, not whether remembered facts change what a tool-using agent does. We present MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and harness that measures the marginal utility of memory for task-executing agents under explicit cost accounting. MERIT provides episodic tool-use tasks in three domains whose dependence on earlier-episode facts is verified by an automated leak check; a difficulty ladder ending in updated-fact recall; controlled memory corruption; and full token and dollar metering of every memory operation. Across 23,440 scored episodes ($42.57), a two-generation pilot on gpt-4.1-mini and a preregistered 3-model x 3-seed grid (GPT-4.1, Claude Haiku 4.5; memory side held fixed), memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00. On updated facts, embedding retrieval collapses unpredictably (0.30-0.95 across models; max seed gap 0.45), and agents act on a correctly retrieved value only 55% of the time, while update-on-write stores (a structured fact store and, notably, LLM summarization) remain at 0.70-1.00; the hybrid is worse than the fact store alone. A latest-generation spot-check (Claude Sonnet 5, gated on a clean full-replay control) reproduces the pattern. Swapping a memory's implementation moves task success by up to 60 points, and full replay is never economical: the best condition per domain delivers 2.7-3.9x its marginal utility per dollar. We release the benchmark, harness, and all traces.

1 Introduction

MERIT addresses three gaps in agent-memory evaluation: conversational recall does not test tool-use decisions, memory harm is undermeasured, and utility is rarely related to cost. It introduces a leak-checked, corruption-aware task benchmark with new utilization and cost metrics, and validates it across broad experimental grids.

Motivation: Conversational QA benchmarks do not establish whether remembered facts improve tool calls and task-execution utility.Production agents must use facts such as customer IDs, refund amounts, or configuration fixes in actions.
Motivation: Existing evaluations undermeasure stale-memory harm and agents’ failure to use correct retrieved facts during task execution.MERIT targets confidently wrong actions from stale records and the failure to act on a fact that is present in memory.
Contributions: MERIT measures cost-adjusted marginal utility, defined as the change in success probability per marginal dollar across memory architectures.This addresses the practical question of whether memory improves success enough to justify its per-task cost.
Contributions: The benchmark uses three-domain episodic tool-use tasks with controllable dependency, a one-fact-to-updated-fact difficulty ladder, and an automated leak check.The leak check verifies that probe inputs and tools do not expose or re-derive the gold fact.
Contributions: Controlled stale, contradictory, and distractor records are injected at known rates to measure Stale-Memory Harm.The corruption conditions have ground-truth flags, enabling controlled measurement rather than observational diagnosis.
Validation: The study covers 23,440 scored episodes and shows that updated-fact retrieval is model- and seed-unstable while update-on-write memories remain uniformly robust.The pilot and preregistered replication use multiple memory conditions, domains, difficulty tiers, and agent models.
Release: The benchmark, harness, and all traces are released with a deterministic $0 mock-model mode for validating the full pipeline.The release supports reproducibility without API cost.

2 Related Work

Prior memory systems and benchmarks largely emphasize conversational recall or within-episode tool competence. MERIT extends tool-use evaluation across episodes while measuring reliability risks such as stale-memory harm.

Memory systems: Memory systems such as MemGPT, Mem0, Zep, A-Mem, and rolling summaries are primarily evaluated through conversational QA.Their designs span OS-style paging, structured fact consolidation, temporal graphs, dynamic notes, and summarization.
Memory benchmarks: LoCoMo, LongMemEval, BEAM, and HaluMem evaluate dialogue-history reasoning or memory-operation reliability rather than cross-episode task execution.These benchmarks cover long-context QA, temporal reasoning, updates, abstention, hallucination, and consistency.
Tool-use benchmarks: τ-bench evaluates tool-calling agents with database-state checks, but its tasks are single-episode and do not exercise memory across episodes.MERIT adds arcs, planted facts, probes, and leak checks to a similar tool-use environment.
Reliability: Reliability concerns motivate measuring stale-memory harm because confidently wrong actions can be more costly than abstentions.MERIT injects staleness at known rates and measures both success degradation and re-verification before acting.

3 The MERIT Benchmark

MERIT is an episodic, tool-executing benchmark built around controlled cross-session dependencies, updated facts, corruption, and explicit measurement of success, memory use, harm, and cost. Its safeguards and paired statistical design make memory effects testable against identical task instances.

Task construction: MERIT uses mutable-world episodes grouped into 4–6-episode arcs containing plant, update, probe, and independent episodes.Probe success requires a prior fact that is absent from the current episode’s inputs.
Validity safeguards: Leak checks exclude gold values from probe inputs and initial world state, while delta scoring requires the checker predicate to change during the probe episode.These safeguards prevent inherited world state or hidden inputs from producing false dependent-task success.
Task domains: The benchmark spans customer support, IT operations, and personal assistant domains with domain-specific tools and cross-session facts.Examples include refunds and addresses, configuration fixes and rollback versions, and calendar or preference information.
Difficulty ladder: Its difficulty ladder progresses from one-fact probes to multi-fact composition and updated-fact probes requiring the latest value.The hard tier distinguishes update-on-write stores from memories that accumulate replay or retrieval records.
Memory conditions: The benchmark compares no memory, full replay, retrieval, rolling summaries, structured fact storage, and a hybrid under a shared agent scaffold.Conditions use the same write/read interface, tools, prompts apart from the memory block, and temperature-zero decoding.
Metrics: TSR measures programmatic task success, while MUR traces whether every retrieved gold value appears in executed tool-call arguments.The tracer agrees with blinded human judgment on 93% of audited items, with conservative undercounting of utilization.
Metrics: SMH compares clean and corrupted TSR, and CAMU relates the TSR gain over no memory to the marginal cost over no memory.The harness meters tokens and dollars and reports break-even task value.
Statistical methodology: Statistical comparisons use identical task instances, arc-clustered paired bootstrap confidence intervals, and Holm–Bonferroni correction within preregistered hypothesis families.The procedure uses 10,000 bootstrap resamples and two-sided tests.

4 Experimental Setup

The experiments combine a gpt-4.1-mini pilot with a preregistered multi-model grid using real memory implementations across three domains and three difficulty tiers. Results establish a leak-verified no-memory floor and show distinct behavior for updated facts across memory architectures.

Pilot: The pilot evaluates 9,940 scored episodes at a total API cost of $9.12 across starter and real memory implementations.Both generations use gpt-4.1-mini and the same domain-by-difficulty-by-condition structure.
Full grid: The preregistered full grid evaluates 13,500 further episodes at $33.45 across gpt-4.1-mini, GPT-4.1, and Claude Haiku 4.5.The memory side is held fixed to gpt-4.1-mini with text-embedding-3-small, so the agent model is the varying factor.
Scale: The combined study contains 23,440 scored episodes and costs $42.57.This total combines the pilot and preregistered full grid.
Reported results: The setup reports dependent-task TSR by difficulty tier and memory condition for real implementations across D1, D2, and D3.Table 1 organizes the dependent-task results by tier and condition, with domain columns for the three domains.
Reported results: The hard updated-fact tier separates overwrite-based memories, chronological replay, and retrieval-based memories.Figure 1 identifies the fact store and LLM summarization as overwrite-based conditions and contrasts them with retrieval conditions.

5 Results

Results show that memory improves dependent task execution, but outcomes depend strongly on architecture, implementation, utilization, model, and cost. Updated-fact tasks expose retrieval instability, while update-on-write approaches are more robust and economical.

Baseline and memory utility: 0.00 dependent-task TSR is the leak-verified C0 floor, while memory conditions add +0.55 to +1.00 at easy difficulty.C0 scores 0.83–1.00 on independent tasks, confirming that the dependent-task floor reflects missing memory rather than unsolvable tasks.
Updated-fact performance: 0.50–0.80 hard-tier TSR makes the hybrid worse than the fact store alone at 0.75–1.00 across all three domains.The retrieval component re-imports staleness that the fact store had eliminated.
Implementation sensitivity: 0.00 / 0.15 / 0.00 became 1.00 / 0.70 / 1.00 on hard when C3 changed from truncation to LLM summarization.The result shows that implementation quality can determine whether a memory architecture retains updated facts.
Implementation sensitivity: 0.40 medium-tier TSR in D2 followed the C4 switch from pattern-based to LLM extraction, down from 1.00.The generic extractor missed IT-operations facts that domain-specific patterns caught, while D1 and D3 roughly held.
Memory utilization: 55 correctly retrieved hard-tier values led to only 30 actions, yielding Ignore Rate 0.45; embedding upgrades barely changed this utilization gap.Even full replay ignored up to 0.50 of held facts in multi-fact medium-tier episodes.
Updated-fact performance: 0.35–0.70 hard-tier TSR for embedding retrieval contrasts with 0.75–1.00 for the structured fact store and 0.80–1.00 for LLM summarization.Retrieval exposes stale and fresh values without a recency signal, whereas update-on-write stores overwrite prior values.
Robustness and generality: 0.30–0.95 hard-tier TSR across models and a maximum seed gap of 0.45 make embedding retrieval less predictable than update-on-write memories.The dissociation is significant across models, with C3−C2 of +0.44, +0.38, and +0.28 for gpt-4.1-mini, GPT-4.1, and Haiku 4.5.

6 Discussion

MERIT’s discussion recommends overwrite-based memory for revised facts, warns that implementation and presentation materially affect outcomes, and emphasizes leak controls for task-execution evaluation.

2.7× tokens make full replay a strong accuracy baseline but a poor cost choice, while multi-fact composition remains a weakness.The guidance recommends measuring cost alongside task success rather than treating replay accuracy as sufficient.
Retrieval’s accumulation of stale records motivates overwrite-based memory for revised operational facts, but hybrids still require direct measurement.The recommendation covers structured fact stores and LLM summarization, while warning that a hybrid may not inherit the better component’s behavior.
Stale and fresh values co-occurring without provenance or timestamps can make agents average, ask, or select the stale value.The traces connect this failure to retrieval presentation rather than storage alone.
Up to 60 points of movement from swapping extraction implementations makes the write path a first-class memory risk.The paper also identifies extraction quality as a cost paid through engineering effort or metered tokens.
Leak checks must cover world state as well as prompts because eager agents and cross-task value collisions created task-execution leak channels.The safeguards used delta scoring and arc-wide uniqueness to address these channels.

7 Limitations and Threats to Validity

The paper identifies important validity boundaries: the automated success checker has limited independent validation, the benchmark spans only three synthetic domains, and frontier-model behavior can conflate memory failure with policy prudence.

Construct validity: κ=0.63 validation against a single human annotator leaves the MUR string-containment checker without independent double annotation.The authors characterize this as a construct limitation because programmatic checkers may not capture all real-world success notions.
Architecture-dependent reliability: Embedding retrieval collapses somewhere for every model, whereas LLM summarization is robust across the hard tier, but the collapse location is model-idiosyncratic.Figure 6 encodes models by bar hatch, conditions by color, and seed variability with min–max whiskers.
External validity: Three synthetic domains, scripted users, 10 arcs per cell, and one representative implementation per memory family constrain external validity.LLM paraphrase mode mitigates phrasing overfit on D1, while implementation sensitivity remains consequential.
Frontier-model diagnostics: C1 full replay is required to distinguish memory use from policy prudence in frontier-model comparisons.Claude Opus 4.8 quoted retrieved values but declined to act, whereas Claude Sonnet 5 passed the control cleanly and reproduced the grid pattern.
Frontier-model diagnostics: A single diagnostic cell with one domain, 50 episodes, and two frontier models limits conclusions about provenance skepticism across domains, prompts, and model families.The paper treats this behavior as a distinct, context-dependent axis rather than a generic memory failure.

8 Conclusion

MERIT reframes memory evaluation around whether recall changes tool-using behavior, how it fails, and what it costs. Its results show large differences among architectures and implementations, while the released benchmark, harness, traces, and preregistration support further evaluation.

Conclusion: MERIT reframes agent-memory evaluation from recall alone to behavioral change, cost, and failure modes.The conclusion contrasts conversational recall with action-level evaluation.
Conclusion: 0.70 TSR separates architectures on superseded facts, with C3 at 1.00 versus C2 at 0.30 for Haiku 4.5 on D3-hard.The conclusion reports this as evidence that architectures indistinguishable on single-fact recall diverge when facts are updated.
Conclusion: 55% of correctly retrieved facts are acted upon, showing that retrieval alone does not guarantee task completion.This finding is summarized alongside implementation and cost differences.
Conclusion: 2.7–3.9× lower marginal utility per dollar makes the most accurate memory less economical than the most efficient one.The paper frames cost-adjusted marginal utility as a central practical evaluation axis.
Conclusion: The benchmark, harness, traces, and preregistration are public for memory-system authors to evaluate against MERIT.The release is presented as an invitation to broader evaluation.