PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

Natchanon Pollertlam, Witchayut Kornsuwannawit

arXiv:2603.04814v1cs.CL

TL;DR

Persistent conversational AI must choose between sending full histories to long-context models and maintaining extracted factual memory, a decision that affects both recall and cumulative cost. The paper compares these architectures across three memory benchmarks and a prompt-caching-aware cost model. Long-context GPT-5-mini has higher factual recall on most benchmarks, while memory becomes cheaper after about ten turns at 100k tokens and remains competitive on persona consistency.

  • Problem

    Persistent conversational systems lack a settled accuracy-and-cost basis for choosing between full-history long-context inference and dedicated fact-based memory.

  • Method

    The paper compares Mem0 fact-based memory with long-context LLM inference across three benchmarks and models cumulative API cost with prompt caching.

  • Results

    Long-context GPT-5-mini achieves higher factual recall on most benchmarks, while memory remains competitive on PersonaMem v2; at 100k tokens, memory becomes cheaper after approximately ten turns.

  • Takeaways & Limitations

    Architecture choice should depend on query type, expected interaction count, and required accuracy, with memory favored economically for repeated long-context use.

  • Takeaways & Limitations

    The findings characterize flat-typed fact extraction specifically and should not be generalized to memory-augmented generation as a category.

Abstract

from arXiv · show

Persistent conversational AI systems face a choice between passing full conversation histories to a long-context large language model (LLM) and maintaining a dedicated memory system that extracts and retrieves structured facts. We compare a fact-based memory system built on the Mem0 framework against long-context LLM inference on three memory-centric benchmarks - LongMemEval, LoCoMo, and PersonaMemv2 - and evaluate both architectures on accuracy and cumulative API cost. Long-context GPT-5-mini achieves higher factual recall on LongMemEval and LoCoMo, while the memory system is competitive on PersonaMemv2, where persona consistency depends on stable, factual attributes suited to flat-typed extraction. We construct a cost model that incorporates prompt caching and show that the two architectures have structurally different cost profiles: long-context inference incurs a per-turn charge that grows with context length even under caching, while the memory system's per-turn read cost remains roughly fixed after a one-time write phase. At a context length of 100k tokens, the memory system becomes cheaper after approximately ten interaction turns, with the break-even point decreasing as context length grows. These results characterize the accuracy-cost trade-off between the two approaches and provide a concrete criterion for selecting between them in production deployments.

1 Introduction

The paper compares long-context inference with fact-based memory for persistent conversational AI, measuring accuracy and cached API cost. Long-context models generally recall facts better, while memory has a distinct cost profile and becomes economical as interactions accumulate.

  • 1 Introduction: The paper compares Mem0-based fact memory with long-context LLM inference across three memory-centric benchmarks and cumulative API cost.The cost model incorporates prompt caching and interaction turns.
  • 1 Introduction: Long-context LLMs achieve higher factual recall on most datasets, while the memory system remains competitive on persona-consistency tasks.The comparison spans varying context lengths and question types.
  • 1 Introduction: Cached long-context inference still has a growing per-turn cost, unlike the memory system’s roughly fixed per-turn read cost after writing.The architectures therefore have structurally different cost profiles.
  • 1 Introduction: At 100k tokens, the memory system becomes cheaper after approximately ten interaction turns, and this threshold decreases as context length grows.The threshold is the break-even point for cumulative cost.

2 Related Work

Related work positions the study between long-context LLMs and increasingly structured memory systems. It uses flat fact extraction as a controlled baseline while addressing accuracy, retrieval, and deployment-cost concerns.

  • 2 Related Work: Long-context models can process histories of one million tokens or more, but performance may degrade when relevant information appears in the middle of prompts.Prior work also studies prompt compression and model cascading.
  • 2 Related Work: Conversational memory research distinguishes working, factual, and episodic memory, with systems ranging from retrieval augmentation to tiered and temporally structured storage.The cited systems represent increasingly structured approaches to persistent memory.
  • 2 Related Work: This work treats long-context inference with prompt caching and flat-typed Mem0 extraction as the two candidate architectures for persistent conversational memory.The comparison is against long-context inference rather than other memory architectures.
  • 2 Related Work: Prior long-context benchmarks document lost-in-the-middle degradation, discrepancies between nominal and effective context size, and retrieval challenges in very long sequences.These findings motivate careful evaluation of long-context memory behavior.

3 Methodology

The methodology compares a customized Mem0 memory pipeline with long-context models that directly pass raw histories to the answer generator. The memory pipeline extracts, embeds, stores, and retrieves flat facts before reasoning.

  • 3 Methodology: The study evaluates both architectures through an accuracy-and-economic-feasibility framework covering datasets, metrics, and cumulative cost.The methodology explicitly compares performance and economic feasibility.
  • 3 Methodology: The Mem0 baseline segments dialogues chronologically, extracts atomic flat-typed facts, embeds them, and stores them in a vector database.The pipeline is designed to preserve temporal order and support vector retrieval.
  • 3 Methodology: For each query, the memory system performs vector search to retrieve relevant facts, which GPT-5-mini then uses as context for reasoning.The retrieval mechanism uses top_k=20.
  • 3 Methodology: The long-context baseline skips memory extraction and indexing, passing the raw conversation history directly to the answer-generation LLM.The setup evaluates GPT-5-mini and GPT-OSS-120B with timestamps included in the conversation format.

3.3 Experimental Datasets

The experiments use three benchmarks with different context lengths and memory demands, evaluate answer correctness with an LLM judge, and model cumulative costs over repeated turns. The economic analysis separates long-context caching from memory writing and reading.

  • 3.3 Experimental Datasets: The benchmarks are LongMemEval, LoCoMo, and PersonaMem v2, covering long-context retrieval, complex reasoning, and persona consistency.Their conversation and question token statistics are summarized separately.
  • 3.3 Experimental Datasets: Accuracy is the percentage of questions judged correct relative to the dataset’s total questions.GPT-5-mini evaluates generated answers using a three-vote majority protocol.
  • 3.3 Experimental Datasets: The economic evaluation compares cumulative memory and long-context costs over N interaction turns, including prompt caching.The analysis considers N ∈ {1, 5, 10, 20}.
  • 3.3 Experimental Datasets: Long-context cost includes a full-price first turn and discounted subsequent turns using the same context.The cached-turn rate applies to repeated queries with a shared context.
  • 3.3 Experimental Datasets: Memory cost combines one-time fact extraction and embedding with turn-by-turn processing of retrieved facts and the current query.These components are represented as Cwrite and Cread.
  • 3.3 Experimental Datasets: The break-even point is the turn count at which CMem becomes lower than CLC, evaluated across context lengths.This measures economic scalability as data volume changes.

3.6 Experimental Environment

The experimental environment routes model and embedding calls through OpenRouter and uses PostgreSQL 17.5 with pgvector for vector storage.

  • All LLM and embedding API calls were routed through OpenRouter.
  • The vector database used pgvector running on PostgreSQL 17.5.
  • Appendix A provides the full prompt templates for fact extraction and LLM-as-a-judge evaluation.

4 Experiments and Results

Long-context GPT-5-mini delivers higher factual recall on LoCoMo and LongMemEval, while the memory system remains competitive on PersonaMem v2. Cost advantages shift with interaction volume: memory becomes cheaper after roughly ten turns at about 100k tokens because its per-turn read cost stays low while cached long-context costs continue growing.

  • Accuracy Results: 35.2 and 33.4 percentage points are GPT-5-mini’s accuracy advantages over the memory system on LoCoMo and LongMemEval, respectively.
  • Accuracy Results: 7.3 percentage points is the smaller GPT-5-mini accuracy advantage on PersonaMem v2, where the memory system marginally outperforms LC GPT-OSS-120B.
  • Accuracy Results: 2,909 tokens per user is the average size of the extracted fact representation from roughly 101,600-token conversations, so compression inevitably loses some information.
  • Operational Cost Breakdown: $21.76 is the one-time fact-extraction cost across 500 LongMemEval conversations, while retrieved-fact answer generation costs roughly $0.0013 per query.
  • Break-even Sensitivity Analysis: Approximately 10 turns is where long-context cost overtakes memory cost at 101,601 tokens; at N = 20, the memory system is 26% cheaper.
  • Break-even Sensitivity Analysis: Cached long-context cost still grows with context length on every turn, whereas memory cost grows at the much smaller per-turn read rate.
  • Break-even Sensitivity Analysis: 13 to 9 turns is the break-even decline as context length increases from 30k to 500k tokens.
  • Summary: The accuracy–cost trade-off favors long-context recall across most benchmarks but favors memory economics for longer interactions, while memory remains competitive on persona consistency.

5 Discussion

The accuracy–cost trade-off depends on both task type and deployment scale: long-context inference preserves more information for complex factual recall, while flat-typed memory is competitive for persona consistency and becomes cheaper as repeated interactions and context length grow. The study’s conclusions are bounded by its single memory baseline, benchmark coverage, evaluation protocol, pricing assumptions, and static-memory setup.

  • Accuracy trade-offs: A roughly 35:1 compression ratio—2,909 retrieved tokens from mean 101,601-token contexts—helps explain the memory system’s factual-recall disadvantage.Flat extraction can irretrievably lose temporal markers, coreferences, and ephemeral details needed for multi-hop questions.
  • Accuracy trade-offs: 62.48% versus 60.50% on PersonaMem v2 shows the memory system can narrowly outperform LC GPT-OSS-120B on stable persona attributes.Persona preferences, personal history, and habitual behaviors are suited to discrete flat-typed facts.
  • Accuracy trade-offs: GPT-OSS-120B scores 10–15 percentage points below GPT-5-mini on LoCoMo and PersonaMem v2, showing that answer-model capability also affects accuracy.On LongMemEval, its 48.20% score is nearly identical to the memory system’s 49.00%, reflecting weaker performance on complex long histories.
  • Cost trade-offs: $0.0013 per query keeps the memory system’s read cost roughly fixed, while cached long-context inference still grows linearly with context length.The comparison assumes a 90% cached-input discount from the second turn onward.
  • Limitations: The study evaluates only Mem0’s flat-typed extraction, so its accuracy–cost findings should not be generalized to memory-augmented generation as a category.More structured temporal or hierarchical memory systems may recover information lost by flat extraction.
  • Limitations: The benchmarks omit real-time knowledge, structured-data reasoning, and compositional multi-document evidence chains.They share a focus on persistent user-specific information across multi-session dialogues.
  • Limitations: The GPT-5-mini judge may introduce family bias or leniency toward verbose near-misses despite the three-vote consensus protocol.These evaluator effects can add variance to the reported accuracy comparisons.
  • Cost trade-offs: At 100k tokens, memory becomes cheaper after approximately ten turns, and at 500k tokens the break-even threshold drops to nine turns.The heatmap’s boundary shifts toward fewer turns as context length increases, even after accounting for the memory write phase.

6 Conclusions and Future Work

The paper finds that choosing between Mem0 memory and long-context inference depends on accuracy requirements, query type, and repeated interaction count. Long-context inference is more accurate on factual-recall tasks, while memory reduces cumulative cost for sufficiently long sessions and remains competitive on persona-consistency tasks.

  • Conclusions: Long-context GPT-5-mini significantly outperforms Mem0 on LoCoMo and LongMemEval, while Mem0 remains competitive on PersonaMem v2.PersonaMem v2 contains information suited to flat fact extraction.
  • Conclusions: At 100k tokens, the two architectures reach cost break-even at approximately ten interaction turns, with the break-even point decreasing as context length grows.
  • Conclusions: For more than approximately ten queries against the same conversational context, Mem0 reduces cumulative inference cost while remaining competitive on persona-consistent tasks.
  • Conclusions: Long-context inference remains preferable for single-session or precision-critical workloads.
  • Future Work: Future work should evaluate richer memory architectures and tasks involving real-time knowledge, structured-data reasoning, and compositional multi-document inference.

A Prompt Templates

The appendix documents the prompt templates and extraction instructions used to construct and judge Mem0 memories. The extraction guidance emphasizes self-contained personal facts, while the evaluation template frames grading against gold answers.

  • A.1 Fact Extraction: Memory Extraction Instructions: The extraction prompt asks the system to generate personal memories following specified guidelines.
  • A.1 Fact Extraction: Memory Extraction Instructions: Each memory should be self-contained and include complete context.
  • A.1 Fact Extraction: Memory Extraction Instructions: The requested memory content includes a person’s name, personal details, emotional states, ongoing journeys, future plans, and specific event dates.
  • A.1 Fact Extraction: Memory Extraction Instructions: Figure 2 presents custom instructions passed to the Mem0 add stage for memory extraction.
  • A.1 Fact Extraction: Memory Extraction Instructions: Figure 2 is identified as the custom-instruction figure for the Mem0 add stage.
  • A.2 LLM-as-a-Judge Grading Prompt: The evaluation appendix introduces system and user prompts used for LLM-as-a-judge grading.

B Model Parameters

The model-parameter appendix specifies different inference controls for the GPT-5 series and GPT-OSS-120B.

  • B Model Parameters: GPT-5-nano and GPT-5-mini use reasoning_effort rather than temperature or max_tokens through the API.The GPT-OSS-120B model uses standard temperature and max-token settings.

C Experimental Environment

The experimental environment routes model and embedding calls through OpenRouter, uses PostgreSQL with pgvector for vector storage, and documents evaluation prompts and model parameters. Pricing inputs come from the OpenAI API pricing page.

  • C Experimental Environment: All LLM completions and embedding calls were routed through OpenRouter as a unified API provider.
  • C Experimental Environment: Vector storage used the pgvector extension running on PostgreSQL 17 as a local service alongside the evaluation pipeline.
  • C Experimental Environment: The appendix includes the system prompt used for LLM-as-a-judge evaluation.
  • C Experimental Environment: Table 7 reports intended inference parameters for GPT-5-nano, GPT-5-mini, and GPT-OSS-120B.
  • C Experimental Environment: Model pricing for the cost-performance analysis was sourced from the OpenAI API pricing page.
Loading 2603.04814v1…