Source-linked AI summary
Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents
Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall, Michael Montero, Adam B. Struck
TL;DR
Persistent memory is needed for agents to maintain context across interactions, but raw-history prompting increases token use and can degrade effective retrieval. Memori structures dialogue into semantic triples and conversation summaries, achieving strong LoCoMo reasoning performance with a compact context footprint. The paper’s category results also show remaining challenges in temporal and open-domain reasoning.
Problem
Persistent memory must support cross-session context without relying on increasingly large raw conversation prompts that raise cost and instability.
Method
Memori uses Advanced Augmentation to distill dialogue into structured semantic triples and conversation summaries for retrieval and downstream reasoning.
Results
Memori achieves strong reasoning performance with a small prompt context footprint, including particularly strong temporal and single-hop results among retrieval-based systems.
Takeaways & Limitations
The results support structuring memory representations rather than simply increasing context as a practical direction for persistent LLM-agent memory.
Takeaways & Limitations
Open-domain reasoning remains challenging because improving it typically requires retrieving larger text chunks, conflicting with Memori’s goal of minimizing context tokens.
Abstract
from arXiv · showhide
As large language models (LLMs) evolve into autonomous agents, persistent memory at the API layer is essential for enabling context-aware behavior across LLMs and multi-session interactions. Existing approaches force vendor lock-in and rely on injecting large volumes of raw conversation into prompts, leading to high token costs and degraded performance. We introduce Memori, an LLM-agnostic persistent memory layer that treats memory as a data structuring problem. Its Advanced Augmentation pipeline converts unstructured dialogue into compact semantic triples and conversation summaries, enabling precise retrieval and coherent reasoning. Evaluated on the LoCoMo benchmark, Memori achieves 81.95% accuracy, outperforming existing memory systems while using only 1,294 tokens per query (~5% of full context). This results in substantial cost reductions, including 67% fewer tokens than competing approaches and over 20x savings compared to full-context methods. These results show that effective memory in LLM agents depends on structured representations instead of larger context windows, enabling scalable and cost-efficient deployment.
1 Introduction
Persistent memory is foundational for context-aware agents because LLM parameters cannot be updated during deployment. Memori frames memory as a structuring problem, distilling conversations into efficient representations for retrieval and reasoning.
- External memory enables agents to persist information, adapt to users, and learn across interactions because deployed LLM parameters cannot update in real time.
- Persistent memory supports continuity in assistants and other applications that must retain and reason over historical information across interactions.
- Naively injecting past interactions expands context windows, increasing cost and instability while making relevant information harder for models to use effectively.
- Memori treats memory as a structuring problem: noisy dialogue must become representations that are efficient to retrieve and effective for downstream reasoning.
- Memori’s Advanced Augmentation incrementally distills conversational data into structured representations, aiming to preserve reasoning performance while reducing prompt tokens.
2 System Architecture
Memori is a decoupled memory layer whose Advanced Augmentation pipeline converts noisy dialogue into linked semantic triples and conversation summaries. This dual representation supports precise retrieval while retaining narrative context for temporal and complex reasoning.
- Memori sits between application logic and the underlying LLM, wrapping existing clients through a lightweight SDK to manage memory.
- Raw transcript retrieval can produce false positives, contradictory context, and inflated generation-time token consumption because conversations contain substantial noise.
- Advanced Augmentation filters raw dialogue into searchable memory assets, shifting memory from text storage toward an organized knowledge base.
- Advanced Augmentation: The pipeline extracts concrete facts, preferences, constraints, and evolving attributes as semantic triples linked to their source conversations.
- Advanced Augmentation: Triples provide precise facts, while conversation summaries preserve narrative flow and temporal context for complex reasoning.
3 Experiments
The experiments evaluate Memori’s Advanced Augmentation on LoCoMo, a multi-session benchmark requiring state tracking, temporal reasoning, and retrieval of subtle preferences. Conversations are transformed into triples and summaries for semantic retrieval and question answering.
- Dataset: LoCoMo evaluates whether agents can track, retain, and synthesize information across extensive multi-session chat histories.
- Dataset: The benchmark includes complex state tracking, temporal reasoning, and subtle preference retrieval from noisy conversational logs.
- Evaluation scope: The evaluation excluded LoCoMo’s adversarial category to maintain comparability with other published benchmark results.
- Evaluation setup: All LoCoMo conversation sections were processed into semantic triples and conversation-level summaries for benchmarking.
- Evaluation setup: Memori used embedded triples and local FAISS indexing to support efficient semantic retrieval for benchmark question answering.
- Evaluation setup: Each question was answered with GPT-4.1-mini using retrieved triples and corresponding summaries, with hybrid cosine-similarity and BM25 retrieval.
3.4 Performance Metrics: LLM-as-a-Judge
The evaluation uses an LLM-as-a-Judge framework and compares reasoning quality with context consumption across memory systems and a full-context ceiling. It emphasizes the tradeoff between answer accuracy and prompt-token efficiency.
- LLM-as-a-Judge: GPT-4.1-mini evaluates generated responses by comparing each query, ground-truth answer, and response.
- Efficiency evaluation: The experiments measure system efficiency as a function of context consumption against traditional architectures such as standard RAG.
- Token efficiency: Prompt-injected token count is treated as the primary operational-cost metric, contrasting concise structured memories with large raw-text or full-history contexts.
- Benchmark comparison: The benchmark compares factual accuracy and reasoning capabilities for Memori, established memory baselines, and a full-context ceiling.
- Benchmark comparison: Memori’s reported performance values were averaged across three evaluation rounds, while baseline and full-context results came from prior work.
3.7 Overall Performance
Memori achieves strong overall and category-level accuracy while narrowing the gap to Full-Context retrieval. Its strengths are clearest in single-hop and multi-hop reasoning, while temporal and open-domain reasoning remain more challenging.
- 87.52% was the highest score for Full-Context, but its production use is constrained by token costs, context expansion, and context degradation.Memori is evaluated against this Full-Context ceiling rather than replacing it as the accuracy maximum.
- 81.95% overall accuracy made Memori the leading retrieval-based system, ahead of Zep at 79.09%, LangMem at 78.05%, and Mem0 at 62.47%.Overall scores are weighted by the number of questions in each LoCoMo reasoning category.
- Single-Hop Reasoning: 87.87% Single-Hop accuracy was Memori’s strongest category result, exceeding LangMem at 74.47% and Zep at 79.43%.The result reflects strong direct fact retrieval from structured memory.
- Temporal Reasoning: 80.37% Temporal accuracy exceeded Mem0 at 66.47% but trailed LangMem at 86.92% and Zep at 83.33%.The results indicate that temporal reasoning remains an area needing improvement.
- Open-Domain Reasoning: 63.54% Open-Domain accuracy remained challenging across retrieval-based systems, with Memori slightly behind LangMem at 67.71%.Improving this category typically requires larger retrieved text chunks, conflicting with token minimization.
3.9 Token Usage and Cost Efficiency
Memori reduces token use by replacing raw conversational retrieval with structured semantic representations and summaries. This lowers prompt overhead while retaining the reported accuracy and reducing operational costs relative to competing and full-context approaches.
- Structured semantic triples and concise conversation summaries replace noisy raw text retrieval, passing high-signal information to the LLM while minimizing context overhead.The pipeline compresses chat logs into structured representations rather than injecting raw conversational chunks.
- Table 2 compares methods using absolute context tokens and cost per query, with costs calculated from gpt-4.1-mini pricing at $0.8 per 1M tokens.Mem0, Zep, and Full-Context values were retrieved from Chhikara et al. [2025].
- 1,294 tokens per response grounded Memori’s LLM outputs, representing 4.97% of the full conversational context while achieving 81.95% overall accuracy.The reported footprint measures the tokens retrieved and added to the prompt.
- 67% fewer tokens than Zep’s 3,911 tokens per query accompanied Memori’s higher accuracy of 81.95% versus 79.09%.The passage directly links the prompt-size reduction to a comparable reduction in API inference costs.
- Over 20 times more tokens than Memori were required by Full-Context retrieval, which used 26,031 tokens per turn.The full-history approach is described as financially unsustainable for persistent agents.
4 Conclusion
Memori treats persistent memory as a data-structuring problem, using Advanced Augmentation to produce compact representations for accurate retrieval and reasoning. On LoCoMo, this approach combines strong reasoning performance with reduced prompt-token use and improved deployment efficiency.
- Memori addresses context degradation and rising token costs by treating persistent memory as a data-structuring problem.
- Advanced Augmentation transforms noisy conversational logs into compact representations combining semantic triples with conversation-level summaries.
- Memori achieves state-of-the-art performance among retrieval-based systems, with particularly strong gains in temporal and single-hop reasoning.
- A small fraction of the original conversation can ground responses without sacrificing accuracy, demonstrating a minimal context footprint.
- Reducing prompt-injected tokens lowers inference costs and supports sustainable deployment of long-running agents.
- The results frame memory quality of structure, rather than context quantity, as the determinant of performance while supporting accurate cross-session recall at lower operational cost.
A Appendix A
Appendix A presents the results-generation prompt used by the memory assistant. It supplies timestamped memories and summaries, emphasizes temporal interpretation and evidence-based answering, and constrains answers to fewer than 5–6 words.
- The results-generation prompt casts the system as a memory assistant retrieving accurate information from conversation memories.
- The prompt provides timestamped factual triples and timestamped conversation summaries as two complementary information sources.
- The instructions prioritize direct memory evidence, recent information when memories conflict, and conversion of relative time references into specific dates.
- Generated answers are limited to fewer than 5–6 words.
- The prompt directs the assistant to examine relevant memories first, use summaries when necessary, and formulate answers from the available evidence.
B Appendix B
Appendix B specifies the LLM-as-a-Judge prompt for evaluating generated answers against questions and gold answers. It uses permissive topic and time-equivalence criteria and returns a binary CORRECT or WRONG label in JSON.
- The evaluation prompt labels each generated answer as CORRECT or WRONG using a question, gold answer, and generated response.
- Answers are counted as correct when they touch the same topic as the concise gold answer, even if they are longer.
- For time-related questions, answers are correct when they refer to the same date or time period as the gold answer, even with different formats or relative wording.
- The prompt supplies the question, gold answer, and generated answer as the inputs to the final evaluation step.
- The evaluator is instructed to return only a JSON object containing the CORRECT or WRONG label.
C Appendix C
Appendix C documents the alignment between LoCoMo question categories and their evaluation order. Table 3 provides the corresponding category distribution used to organize the benchmark questions.
- Table 3 maps LoCoMo question categories to their corresponding evaluation order.
- The table also reports the number of questions assigned to each category.