Source-linked AI summary
Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents
Ming Wu, Pengyuan Zhu
TL;DR
Long-term agent memory must preserve facts, revisions, relationships, and evidence across fragmented conversations, files, and connected sources, beyond what single-structure systems provide. Agent Zero Memory addresses this with three provenance-aware memory systems and citation-locked, agent-controlled retrieval. It achieves state-of-the-art results on two benchmarks and shows similar accuracy across backbones despite large cost differences.
Problem
Long-term agent memory must recover facts, track revisions, and cite evidence across conversations, files, and connected sources, while single organizing structures have blind spots.
Method
Agent Zero Memory distils sources into an events timeline, entity–event graph, and citation-locked HDM, then uses intent-gated, routed, concurrent agentic searches.
Results
95.60% on LongMemEval and 93.60% on LoCoMo establish state-of-the-art results, while accuracy across eight backbones varies by only 3.4 points despite large cost differences.
Takeaways & Limitations
The results support provenance-aware layered memory as a durable abstraction whose quality is driven more by retrieved memory structure than by any single backbone.
Takeaways & Limitations
Competitor comparisons use the best publicly reported numbers under different harnesses, so they are not perfectly controlled head-to-head trials.
Abstract
from arXiv · showhide
Large language model (LLM) agents need durable, faithful memory of everything a user or organization has said and stored, yet most memory systems commit to a single organizing structure (a fact store, a vector index, or a knowledge graph) and inherit its blind spots. We present Agent Zero Memory, a provenance-aware long-term memory system that distils a user's conversations, files, and connected sources into three parallel memory systems, each capturing a different facet of the same history: an episodic Memory Events timeline that makes when and what changed first-class, an associative entity-event knowledge graph that links people and projects across sessions, and a semantic, curated, citation-locked Hierarchical Documentary Memory (HDM) of durable facts. A retrieval turn runs an intent gate (so self-contained turns add no latency), a source router, and three concurrent agentic searches, one per system, each a tool-using loop over hybrid (embedding + lexical) search under agent-controlled filters; their grounded, cited answers are integrated into one answer with a single confidence. We formalize the reading discipline: every learned item is a provenanced item carrying its origin, timestamp, and evidence pointer, and every answer is read under a citation lock, so it may cite only evidence its reader actually opened; fabrication is structurally excluded and the system abstains rather than guesses. On two public benchmarks the system sets a new state of the art: 95.60% on LongMemEval and 93.60% on LoCoMo, improving over the strongest prior systems by +0.73 and +1.10 points. A controlled study across eight backbone LLMs characterizes the accuracy-cost-latency frontier: accuracy varies by only 3.4 points while per-query cost varies by ~30x, with near-state-of-the-art quality at up to 20x lower cost per query, the signature of memory-driven, rather than model-driven, quality.
1 Introduction
Agent Zero Memory addresses the fragmentation and structural blind spots of long-term agent memory with three complementary, provenance-aware memory systems and citation-locked retrieval. It reports state-of-the-art benchmark performance and a favorable accuracy–cost–latency trade-off across backbone models.
- Motivation: Long-term agent memory must recover facts, track later changes, and cite the specific evidence supporting each version.Existing retrieval mechanisms do not jointly provide these capabilities across conversations, files, and connected tools.
- Architecture: Agent Zero Memory combines a Memory Events timeline, an entity–event knowledge graph, and a curated citation-locked Hierarchical Documentary Memory.The three systems represent temporal changes, cross-session relationships, and durable documentary facts respectively.
- Provenance discipline: The system formalizes provenanced items and citation-locked answers, structurally excluding fabrication and requiring abstention when evidence cannot be cited.Answers may cite only evidence opened by the reader.
- Retrieval: Retrieval uses an intent gate, source router, and concurrent tool-using searches over hybrid embedding and lexical retrieval with agent-controlled filters.Self-contained turns bypass memory, while each search can select filters and open relevant evidence.
- Results: 95.60% on LongMemEval and 93.60% on LoCoMo exceed the strongest reported systems by 0.73 and 1.10 points.These are reported as state-of-the-art results on both public benchmarks.
- Results: 92.2% on LongMemEval is reached by a compact open model at roughly 1/20 the per-query cost of the best configuration.The controlled study spans eight LLM backbones and characterizes accuracy–cost–latency trade-offs.
2 Related Work
Prior long-term memory systems organize information around single structures or concentrated context mechanisms, creating limitations in temporal reasoning, provenance, contradiction handling, or scaling. Agent Zero Memory combines complementary memories with routed, parallel, agent-controlled retrieval to address these axes together.
- Design space: Existing systems differ in organization, update semantics, provenance, read-cost scaling, and shareability, but no single representation serves all question types.The comparison frames these dimensions as central to benchmark performance.
- Context paging: MemGPT pages unstructured text through the context window, lacking temporal and relational organization while consuming model turns during conversations.Its approach removes hard context limits without external infrastructure.
- Fact extraction and update: Fact-extraction systems such as Mem0 keep query context small, but pay extraction and reconciliation costs during ingestion.The cited discussion identifies write-side cost as a scaling concern.
- Update and provenance: Flat or overwrite-based stores can lose history, source traceability, or reliable contradiction handling when memories are updated.Memobase overwrites current values and deletes raw messages, while other systems defer contradiction resolution or risk mislinked version chains.
- Structured memory: Structured approaches improve temporal or associative retrieval but incur write amplification, corpus-growing read costs, lossy distillation, or missing update semantics.The surveyed examples include Zep, HippoRAG, and related graph-based designs.
- How we differ: Agent Zero Memory uses three complementary memories, an intent gate, source routing, and parallel agentic search so only relevant memory is touched.Its retrieval combines dense and lexical methods over structured, provenanced stores with agent-controlled filters.
3 Methodology
Agent Zero Memory builds three provenance-aware, complementary memory systems and retrieves from them through citation-locked, concurrent agentic searches. Its design distinguishes temporal, relational, documentary, and lifecycle-specific memory while abstaining when evidence cannot support an answer.
- Three parallel memory systems: The system maintains an events timeline, an entity–event graph, and a documentary memory as three complementary stores.The timeline supports temporal and update queries, the graph supports multi-hop retrieval, and HDM stores curated durable facts.
- Provenance and citation locking: Provenanced items record origin, immutable creation time, and an evidence pointer resolving to raw artifacts.This makes derived memory auditable across inferred, documentary, dialogue, manual, and scanned origins.
- Provenance and citation locking: Citation-locked answers may cite only opened evidence supporting every atomic claim, otherwise the system abstains.The interface structurally excludes fabrication rather than merely discouraging it.
- Memory Build: Memory Build preprocesses and classifies sources, indexes them with embedding and lexical search, then extracts events and documentary entries before connecting the graph.The four-stage background pipeline creates complementary views of chats, files, and agent sessions.
- Memory Build: The Memory Events timeline preserves what happened, when it happened, how it unfolded, and which source supports it.Events are the primary recall unit, with raw chats or files opened when additional detail is needed.
- Memory Build: HDM organizes curated profile and preference knowledge into six categories and three detail levels, opening fuller text only when needed.Its entries derive deterministically from curated notes, record their source, and remain citation-locked.
- Memory Injection: Retrieval uses an intent gate, source router, and three concurrent tool-using searches with dense-plus-lexical retrieval and agent-controlled filters.Searches open raw chunks on demand, then integrate partial cited answers into one confidence-scored response; temporal-order requests bypass reranking to preserve chronology.
- Continual learning: Working-memory records useful to later tasks are promoted into persistent experiential memory, allowing lessons learned by one agent to become available to others.This continual-learning loop distinguishes factual, experiential, and session-bound working memory by source and lifecycle.
4 Experimental Setup
The evaluation tests Agent Zero Memory on two public long-term conversational-memory benchmarks and compares it with leading graph- and vector-based systems. A controlled study varies only the inference backbone across eight models while keeping the memory system fixed.
- Benchmarks: LongMemEval contains 500 questions spanning six question types and an abstention condition for answers never stated.Knowledge-update and temporal questions test distinguishing superseded values from corrections and reasoning about when facts held.
- Benchmarks: LoCoMo contains 1,540 questions over multi-hundred-turn, multi-session dialogues across single-hop, multi-hop, temporal, and open-domain categories.Multi-hop and temporal questions require composing evidence from widely separated turns.
- Evaluation procedure: Both benchmarks build the three memory systems from session histories and answer held-out questions through the intent-gated, source-routed, concurrent-search pipeline.The evaluation therefore applies the complete Memory Build and Memory Injection procedure.
- Baselines: The comparison includes Zep, Mem0, Mastra, Hindsight, EmergenceMem, Supermemory, ByteRover, and Memobase using their best publicly reported figures.These baselines include graph- and vector-based memory systems.
- Controlled backbone study: The controlled backbone study varies only the LLM driving inference and reasoning while fixing the memory stores, indexes, retriever, and control logic.Eight backbones span three vendors and a wide capability and price range.
- Implementation: The stores and indexes run on PostgreSQL, with pgvector dense retrieval over 3072-dimensional embeddings and BM25 plus fuzzy lexical matching.Each agentic search retrieves candidates from both channels and fuses them.
5 Results
Agent Zero Memory achieves state-of-the-art accuracy on both long-term memory benchmarks, while backbone choice exposes substantial cost and latency trade-offs. Retrieval ablations further show that hybrid embedding and lexical search contributes complementary gains.
- Main results: 95.60% on LongMemEval and 93.60% on LoCoMo establish state-of-the-art results, exceeding strongest reported systems by 0.73 and 1.10 points.
- Backbone study: Accuracy remains within 3.4 points across eight backbones, indicating stable quality under a fixed memory and retrieval substrate.The evaluated range is 92.20% to 95.60%.
- Backbone study: The fastest backbone, glm5.2fast, reaches 93.00% at a median 6.16 s, while deepseek-v4-pro reaches 92.20% at $0.001768 per query.The best-accuracy configuration, gpt-5.5, is neither fastest nor cheapest.
- Operating points and cost efficiency: Per-query cost varies by ∼30× while accuracy varies by only 3.4 points, producing several Pareto-efficient backbone choices.
- Operating points and cost efficiency: Deepseek-v4-pro’s low base price and 88% cache rate make it the cost leader despite emitting the most completion tokens.Cached-input fractions range from 43.6% for gpt-5.5 to 88.2% for deepseek-v4-pro.
- Ablation: hybrid retrieval: Embedding-only, lexical-only, and grep-only retrieval reduce accuracy by 1.2, 1.8, and 1.6 points, respectively, relative to hybrid retrieval.The results support complementary roles for semantic and lexical channels.
6 Discussion
Agent Zero Memory’s retrieval-centered design supports robust quality across backbones while enabling substantial cost and deployment flexibility, though several evaluation comparisons remain imperfectly controlled.
- Quality is retrieval-dominated: 3.4 points: LongMemEval accuracy varies by only 3.4 points across eight backbones, supporting a retrieval- and representation-dominated quality pattern.The authors argue that provenance-tagged evidence allows compact backbones to answer correctly and that retrieval investment transfers across models.
- Deployment economics: ∼1/20 the cost: deepseek-v4-pro comes within 3.4 points of the best LongMemEval system, while glm5.2fast records 6.16 s median latency.The deployment frontier permits trading a small amount of accuracy for large savings.
- Practical deployment: Source references make outputs auditable, while background memory construction and backbone-agnostic stores support incremental building and serving across operating points.Operators can inspect the conversation turn or file supporting a reported answer without rebuilding memory for different backbones.
- Threats to validity: Competitor comparisons are not perfectly controlled because published results use different judges, prompts, retrieval budgets, and evaluation harnesses.The backbone study is controlled with identical memory, retriever, and control logic, varying only the model; reported costs also depend on evaluation-time list prices.
- Limitations: The evaluation follows LLM-judged benchmark protocols, and component-level ablations of the three memories and intent gate remain future work.The paper reports a retrieval-channel ablation but does not yet quantify each major component’s marginal contribution.
7 Conclusion
Agent Zero Memory presents provenance-aware, layered long-term memory built from three complementary systems and concurrent agentic searches. It reports state-of-the-art benchmark performance and a favorable backbone accuracy–cost–latency frontier, while identifying calibration, conflict arbitration, and forgetting as open directions.
- Conclusion: 95.60% and 93.60%: Agent Zero Memory sets new state-of-the-art results on LongMemEval and LoCoMo, respectively.The conclusion also describes robust performance across eight backbone models and a favorable accuracy–cost–latency frontier.
- Future directions: Future work includes component ablations, calibrated confidence probabilities, conflict arbitration across valid fact versions, and graceful forgetting of stale experiential memory.The proposed forgetting mechanism would down-weight stale information rather than delete it, preserving auditability.
- Conclusion: The paper frames provenance-aware, layered memory as a durable abstraction for long-term agent memory rather than relying on one model or per-query pipeline.This conclusion centers the memory substrate as the enduring contribution.