Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

• The evaluation used three tests of remembering user facts, dates, and personal details, and tracked costs with provider prompt-caching discounts included.Source: Executive Summary · p. 1
• The full-history approach was about 33 to 35 percentage points more correct on two tests, while memory performed just as well or slightly better on the test of consistency about personal details.The accuracy comparison varies by test rather than showing one approach as the consistent winner.Source: Executive Summary · p. 1
• At about 100,000 words of conversation history, memory became cheaper after around 10 questions and saved about 26% of total cost by 20 questions.Full-history inference costs less in the first few interactions but grows with each question; memory has an upfront store-building cost followed by a small, steady per-question fee.Source: Executive Summary · p. 1
• For repeated interactions, memory can save money while providing good answers; for short or one-time use when accuracy is paramount, full history is the better choice.The paper offers a way to estimate which approach costs less based on expected interaction frequency.Source: Executive Summary · p. 2

1 Introduction

• Prompt caching discounts shared-prefix tokens, but long-context inference still incurs a per-turn cost proportional to context length; memory has a one-time write phase and smaller subsequent reads.The paper’s cost model tracks cumulative API expenditure as a function of interaction turns.Source: §1 · p. 2
• The study compares Mem0-based fact memory with long-context LLM inference on LongMemEval, LoCoMo, and PersonaMem v2, measuring factual recall and cumulative API cost.The memory approach distills interactions into structured records and retrieves relevant facts at query time.Source: §1 · p. 2
• Long-context LLMs achieve higher factual recall on most datasets, while the memory system remains competitive on persona-consistency tasks.The comparison spans three benchmarks with varying context lengths and question types.Source: §1 · p. 2
• At 100k tokens, the memory system becomes cheaper after approximately ten interaction turns, with the break-even threshold decreasing as context length grows.The break-even point depends on how frequently a user re-engages with the same context.Source: §1 · p. 2

Long-context LLMs

• Long-context models can support million-token histories, but middle-position retrieval, effective context limits, and reasoning can degrade with longer inputs (Liu et al., 2024; Hsieh et al., 2024; Levy et al., 2024).Prompt compression (Jiang et al., 2023) and model cascading (Chen et al., 2023a) have been proposed to reduce long-context inference costs; this work evaluates a prompt-cached long-context model against fact-based memory.Source: §2 · pp. 2–3

Memory systems for conversational agents

• Conversational-agent memory spans working, factual, and episodic forms, while Mem0 extracts flat-typed facts for vector retrieval and serves as this paper’s controlled baseline (Packer et al., 2024; Su et al., 2026; Chhikara et al., 2025).Mem0 separates one-time extraction from query-time reads and reports a 91% reduction in p95 response latency versus full-context processing on LoCoMo (Maharana et al., 2024; Chhikara et al., 2025). Other systems use tiered storage (Packer et al., 2024), hierarchical scenes (Hu et al., 2026), temporal summaries (Su et al., 2026), linked notes (Xu et al., 2025), or temporal knowledge graphs (Rasmussen et al., 2025).Source: §2 · p. 3

Cost efficiency in LLM inference

• Prompt caching discounts cached GPT-5-series input tokens by 90%, and this work applies that discount from turn two onward to estimate when memory becomes cheaper (OpenAI, 2024, 2025a).Prior work also applies caching to static knowledge bases (Chan et al., 2025) and analyzes KV-cache eviction policies, finding standard Least Recently Used can be suboptimal for tail latency in multi-turn conversations (Zhang et al., 2025).Source: §2 · p. 3

Long-context benchmarks and evaluation

• LongMemEval tests five memory abilities on histories of about 115k tokens, LoCoMo spans nearly 2,000 questions across ten dialogues, and PersonaMem v2 tests persona consistency (Wu et al., 2025; Maharana et al., 2024; Jiang et al., 2025).Other long-context evaluations include LongBench (Bai et al., 2023), ∞Bench (Zhang et al., 2024), and Needle-in-a-Haystack (Greg Kamradt, 2023); studies report lost-in-the-middle effects, gaps between nominal and effective context size, and retrieval challenges over very long sequences (Liu et al., 2024; Hsieh et al., 2024; Kuratov et al., 2024).Source: §2 · pp. 3–4

3 Methodology

• The study evaluates a Mem0 memory baseline and long-context LLMs using benchmark datasets, response-accuracy metrics, and a cost-performance analysis.The methodology outlines the memory architecture, datasets, evaluation metrics, and cost-performance framework.Source: §3 · p. 4

3.1 Memory System Baseline (Mem0)

• Mem0 processes dialogues in chronological segments and uses GPT-5-nano to extract atomic, flat-typed facts for vector retrieval.Segments use batch_size=10 and a maximum of 8,000 characters; the extractor uses low reasoning effort.Source: §3.1 · p. 4
• Extracted facts are embedded with text-embedding-3-small into 1536-dimensional vectors and stored in pgvector with an HNSW index.The index uses m=16, ef_construction=64, and cosine similarity.Source: §3.1 · p. 4
• For each query, Mem0 retrieves the top 20 facts by vector search and supplies them to a GPT-5-mini reasoner.The reasoner uses default API parameters.Source: §3.1 · p. 4

3.2 Long-context LLM

• The long-context setup skips memory extraction and passes raw conversation history with timestamps to GPT-5-mini or GPT-OSS-120B for answer generation.Both models use default API parameters.Source: §3.2 · p. 4

3.3 Experimental Datasets

• The evaluation uses LongMemEval for conversations exceeding 100k tokens, LoCoMo for complex long-history reasoning, and PersonaMem v2 for consistent agentic personas.Conversation and question token statistics are summarized in Tables 1 and 2.Source: §3.3 · p. 4

3.4 Evaluation Metrics: LLM-as-a-Judge

• GPT-5-mini judges whether each generated answer aligns with the Golden Answer three times, with majority vote used to reduce stochastic variance and self-preference bias.The judge is selected for its reasoning capabilities and long-context support.Source: §3.4 · p. 4
• Accuracy is the percentage of dataset questions whose generated responses the judge marks correct.The denominator is the total number of questions in the dataset.Source: §3.4 · p. 4

3.5 Cost-Performance Analysis Framework

• The economic evaluation compares cumulative Memory System (CMem) and Long-context LLM (CLC) costs over 1, 5, 10, and 20 interaction turns.The framework also evaluates how the break-even point varies with context length.Source: §3.5 · p. 5

3.5.1 Cost Equations

• For N > 13, prompt caching applies a 90% discount on input tokens in the cost models.For CLC, the first turn reflects the full price of the 100k+ context, while subsequent queries using the same context use the discounted rate.Source: §3.5.1 · p. 5
• The memory system's write cost is a one-time initialization for fact extraction and embedding, using GPT-5-nano and text-embedding-3-small pricing.The read cost is charged turn by turn for processing retrieved facts and the current query, using GPT-5-mini pricing.Source: §3.5.1 · p. 5

3.5.2 Break-even Analysis

• The break-even point is the turn count at which CMem < CLC, assessed across varying context lengths to measure economic scalability sensitivity to data volume.The analysis examines how the break-even point changes with context length.Source: §3.5.2 · p. 5

3.6 Experimental Environment

• All LLM and embedding API calls were routed through OpenRouter, and the vector database was pgvector running on PostgreSQL 17.Source: §3.6 · p. 5
• Appendix A provides the full prompt templates for fact extraction and LLM-as-a-judge evaluation.Source: §3.6 · p. 5

4.1 Accuracy Results

• LC GPT-5-mini achieves the highest accuracy on LoCoMo and LongMemEval, exceeding Mem0 by 35.2 and 33.4 percentage points, respectively; on PersonaMem v2, the gap is 7.3 percentage points.Source: §4.1 · p. 5, Table 3 · p. 6

4.2 Operational Cost Breakdown

• Memory incurred a $21.76 one-time extraction cost and $0.65 for 500 reads, versus $14.79 for 504 full-context GPT-5-mini requests.Memory’s read phase cost roughly $0.0013 per query; after the write phase, its per-query charge stays roughly fixed, while long-context charges scale with the full context on every turn.Source: §4.2 · p. 6

4.3 Cost Comparison by Interaction Volume

• At 101,601 tokens, long-context costs less initially ($0.0265 vs. $0.0450), but memory becomes cheaper at approximately 10 turns and is 26% cheaper at N = 20.The comparison uses prompt caching for long-context inference from turn 2 onward; cached turns still incur an incremental charge proportional to context length.Source: §4.3 · p. 6, Table 4 · p. 7

4.4 Break-even Sensitivity Analysis

• Break-even N falls from 13 to 9 as context grows from 30k to 500k tokens, then differs by at most one turn across 100k, 200k, and 500k.The cost-optimal boundary shifts toward fewer turns as context grows.Source: §4.4 · pp. 6–7, Table 5 · p. 7

4.5 Summary

• The results show an accuracy–cost trade-off: long-context LLMs, particularly GPT-5-mini, achieve higher factual recall across most benchmarks, while memory is economical beyond approximately ten interactions and competitive on PersonaMem v2.The paper identifies PersonaMem v2 as the subset where persona consistency matters most.Source: §4.5 · p. 7

5.1 Interpretation of Results

• At a mean context of 101,601 tokens, flat-typed extraction reduces conversations to 2,909 retrieved tokens per user (roughly 35:1), losing temporal markers, implicit coreferences, and ephemeral details that can impair multi-hop answers.The interpretation attributes the LongMemEval and LoCoMo accuracy gap to raw-history access versus compressed atomic facts.Source: §5.1 · p. 7
• On PersonaMem v2, memory narrowly exceeds LC GPT-OSS-120B (62.48% vs. 60.50%), while GPT-OSS-120B’s weaker results elsewhere show that answer-model capability also affects accuracy.Persona attributes suit flat-typed extraction (Jiang et al., 2025); GPT-OSS-120B scores 10–15 percentage points below GPT-5-mini on LoCoMo and PersonaMem v2, and its near-match to memory on LongMemEval (48.20% vs. 49.00%) reflects underperformance on longer histories.Source: §5.1 · pp. 7–8
• With a 90% input-token caching discount from the second turn, long-context per-turn cost still grows linearly with context length, while memory reads remain roughly fixed at $0.0013 per query.The discussion says this cost-shape difference persists under pricing models that charge proportionally to input token count.Source: §5.1 · p. 8

5.2 Limitations

• This study evaluates only Mem0’s flat-typed extraction baseline (Chhikara et al., 2025), and its benchmarks omit real-time knowledge, structured-data reasoning, and compositional multi-document evidence chains.Other memory designs, including temporal semantic memory (Su et al., 2026) and hierarchical engram-based systems (Hu et al., 2026), may recover lost recall, so findings should not be generalized to memory-augmented generation as a category.Source: §5.2 · pp. 8–9
• The GPT-5-mini LLM-as-a-judge protocol uses three-vote majority consensus but may favor same-family responses and be more lenient on verbose near-misses than human annotators (Zheng et al., 2023; Ye et al., 2024; Chen et al., 2024).The cited work also motivates concern about systematic judge biases (Zheng et al., 2023; Ye et al., 2024).Source: §5.2 · p. 9
• The cost model uses experiment-era OpenAI pricing and a fixed 90% caching discount; actual cache hit rates vary, infrastructure costs are excluded, and GPT-OSS-120B costs are not modeled.Infrastructure costs would increase memory’s absolute cost but are not expected to change the qualitative break-even curve.Source: §5.2 · p. 9
• Because the benchmark ingests all conversations in one write phase, it does not test incremental memory updates when users add or revise facts, leaving their effect on the cost–accuracy trade-off to future work.In active deployments, incremental updates could shift the reported accuracy trade-offs.Source: §5.2 · p. 9

5.3 Practical Implications

• At 100k tokens, memory becomes cheaper after approximately ten turns (nine at 500k), while long-context inference is cheaper and more accurate for single-session use; accuracy trade-offs depend on the task.For repeated interactions, the memory system offers lower cumulative cost with a modest accuracy trade-off, particularly where persona consistency matters.Source: §5.3 · p. 9

6 Conclusions and Future Work

• Long-context GPT-5-mini significantly outperforms Mem0 on LoCoMo and LongMemEval, while Mem0 remains competitive on PersonaMem v2.PersonaMem v2 includes relevant information suited to flat fact extraction.Source: §6 · p. 10
• At 100k tokens, the architectures break even at approximately ten interaction turns, and the break-even point decreases as context length grows.The cost model incorporates prompt caching.Source: §6 · p. 10
• Architecture choice depends on query nature, session interaction count, and application accuracy requirements; Mem0 reduces cumulative inference cost beyond approximately ten queries, while long-context inference suits single-session or precision-critical workloads.Mem0 remains competitive on persona-consistent tasks.Source: §6 · p. 10
• Future work should examine temporal semantic memory (Su et al., 2026) and hierarchical engrambased systems (Hu et al., 2026), and evaluate real-time knowledge, structured-data reasoning, and compositional multi-document inference.Source: §6 · p. 10

A.1 Fact Extraction: Memory Extraction Instructions

• Custom instructions guide Mem0’s add stage in extracting memory from raw conversations (Chhikara et al., 2025).Figure 2 presents the instructions passed to the add stage.Source: §A.1 · p. 14, Figure 2 · p. 14
• The extraction instructions were developed as part of the EverMemOS Mem0 evaluation pipeline (Hu et al., 2026).Source: §A.1 · p. 14
• Figure 2 shows the custom instructions passed to Mem0’s add stage for memory extraction.Source: Figure 2 · p. 14

A.2 LLM-as-a-Judge Grading Prompt

• The judge uses system and user prompts to evaluate generated answers against golden answers.Source: §A.2 · p. 14
• GPT-5-mini serves as the judge in this grading protocol (Singh et al., 2025).Source: §A.2 · p. 14
• The grading protocol was adopted from the EverMemOS evaluation framework (Hu et al., 2026).Source: §A.2 · p. 14

B Model Parameters

• GPT-5-nano and GPT-5-mini use reasoning_effort because their APIs do not accept temperature or max_tokens; GPT-OSS-120B uses standard temperature and max-token settings.Table 7 lists the intended inference parameters for each model role.Source: B Model Parameters · p. 14

Experimental Environment

• LLM completions and embedding calls went through OpenRouter6, while vector storage used pgvector7 on PostgreSQL 17 deployed locally.The database service ran alongside the evaluation pipeline.Source: C Experimental Environment · pp. 14–15
• The LLM-as-a-Judge prompt instructed the grader to determine whether answers matched a gold-standard answer.Figure 3 presents the system and user prompt templates used for judge evaluation.Source: C Experimental Environment · p. 15, Figure 3 · p. 15
• Table 7 specifies inference parameters for GPT-5-nano, GPT-5-mini, and GPT-OSS-120B, including their distinct parameter and snapshot conventions.The first two use reasoning_effort and are pinned to the 2025-08-07 snapshot; GPT-OSS-120B uses temperature and max_tokens without a versioned snapshot. Model pricing came from the OpenAI API pricing page.Source: Table 7 · p. 15, C Experimental Environment · p. 15