PaperlayerView exact revision on arXiv

Source-linked AI summary

MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

Xinle Deng, Ruobin Zhong, Hujin Peng, Xiaoben Lu, Yanzhe Wu, Guang Li, Buqiang Xu, Yunzhi Yao, Jizhan Fang, Haoliang Cao, Junjie Guo, Yuan Yuan, Ziqing Ma, Yuanqiang Yu, Rui Hu, Baohua Dong, Hangcheng Zhu, Ningyu Zhang

arXiv:2605.28732v3cs.CLcs.AIcs.LG

TL;DR

LLM memory systems can fail because errors introduced in persistent memory operations surface much later, making their origins difficult to identify. MemTrace represents execution as operation-variable graphs, benchmarks attribution across representative systems, and uses attribution to improve end-task performance by up to 7.62%.

  • Problem

    Failures in persistent LLM memory systems can originate in earlier construction, update, or deletion operations and surface later, while chronological traces and outcome benchmarks lack causal structure for locating their origins.

  • Method

    MemTrace instruments memory pipelines to build operation-variable execution graphs and attributes failures by tracing dependencies across memory updates, reads, and answer generation.

  • Results

    End-task performance improves by up to 7.62% when fine-grained attribution signals guide automatic system optimization.

  • Takeaways & Limitations

    MemTrace recovers faulty operations and error types with coherent explanations, supporting automatic debugging and optimization of memory systems.

  • Takeaways & Limitations

    MemTraceBench’s scale and diversity remain expandable, and the current formulation focuses on failures whose decisive error set is a singleton.

Abstract

from arXiv · show

Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug. Tracing memory's dynamic evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM memory systems. We propose a novel framework that transforms memory pipelines into executable memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a benchmark collected from representative memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that memory failures are systematic, stemming from operation-level issues like information loss and retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62%. Code will be released at https://github.com/zjunlp/MemTrace.

1 Introduction

MemTrace addresses the underexplored problem of tracing and attributing failures in increasingly complex LLM memory systems, where errors can originate in earlier memory operations and propagate to downstream predictions. It represents execution as an operation-variable graph for fine-grained diagnosis and uses attribution signals to improve end-task performance by up to 7.62%.

  • Motivation: Memory systems enable LLM agents to retain information across interactions, update state over time, and support long-horizon tasks and continual learning.They are used in applications including personalized assistants and coding agents.
  • Problem: Failure attribution is difficult because a memory preference can be correctly stored, later overwritten by an incorrect update, and cause a distant downstream failure.Chronological traces alone cannot reliably connect the downstream failure to its originating operation.
  • Problem: Linear execution logs lack the structure needed to show how memory variables are created, modified, overwritten, propagated, and used in a failed prediction.This distinguishes memory-system diagnosis from failures localized within a stateless agent’s current execution trajectory.
  • Method: MemTrace exposes memory-system execution as a unified operation-variable graph that records operations and associated variables for error tracing and attribution.The framework uses a system-agnostic tracing toolkit and connects variables through shared operations.
  • Results: Up to 7.62%: attribution signals guide automatic system optimization to improve end-task performance.MemTrace also recovers meaningful faulty operations and error types and generates coherent debugging explanations.

2 Tracing and Attributing Errors in Memory Systems

MemTrace attributes memory-system failures by recording execution traces as dependency-aware graphs and analyzing them to identify the earliest decisive faulty operation. Its formulation targets long historical trajectories whose traces can reach tens of megabytes, distinguishing memory systems from shorter single-run agent traces.

  • Attribution Framework: Failure attribution first collects a system execution trace and then analyzes it to localize the failure source.The memory system processes trajectory τ and question q through memory updates UM, memory reads RM, and answer generation Q.
  • Attribution Framework: Source-code instrumentation records variables, applied operations, and dependency relations during updates, reads, and answer generation, producing an execution graph G = (V, O, E).Recorded variables include the input question q and predicted answer ˆa.
  • Problem Definition: Given a failed execution graph and golden answer, the objective is to identify the earliest decisive faulty operation o∗.Operations are sequential and totally ordered by timestamps; a candidate earliest error is faulty while all preceding operations are functionally correct.
  • Memory-System Setting: Memory-system traces differ from prior agent traces because they span long historical trajectories and can grow to tens of megabytes.Prior work often treats traces as relatively short log sequences from a single task run.

3 MemTraceBench Construction

MemTraceBench is constructed to evaluate automatic failure attribution in stateful agents with non-parametric memory. It combines annotated execution traces from four representative memory systems with question–answer pairs from three source datasets, while capturing fine-grained memory evolution and operational dependencies.

  • Benchmark motivation and contents: MemTraceBench addresses the lack of datasets for automatic failure attribution in stateful agents with non-parametric memory.Each example includes a question, golden answer, full system execution trace, and annotated failure information.
  • Limitations: Singleton attribution is insufficient when failures are jointly caused by multiple independent faulty operations; this non-singleton setting and extensions are discussed in Appendix E.The passage identifies multi-operation causality as a limitation of singleton attribution.
  • Benchmark motivation and contents: Annotations identify faulty-operation IDs, error types, and explanations for each benchmark example.The benchmark examples also contain the question, corresponding golden answer, and full execution trace.
  • Data and systems: The benchmark uses question–answer pairs from LoCoMo, LongMemEval, and RealMem across long-context memory, RAG, Mem0, and EverMemOS.These four systems are selected as representative memory systems.
  • Trace collection: Construction collects fine-grained execution graphs that track memory-unit creation, memory evolution, and intermediate-variable dependencies across construction, retrieval, response generation, and evaluation.This requires tracing operational information flow beyond only LLM-call inputs and outputs, despite heterogeneous schemas and code structures.

4 Methodology

MemTrace treats failure attribution in non-parametric memory systems as agentic exploration over executable memory evolution graphs. It combines targeted starting-point retrieval, bounded chronological graph exploration, localized operation views, and a global search variant for weakly structured graphs.

  • Graph exploration: MemTrace frames failure attribution as agentic graph exploration that iteratively inspects local operation subgraphs until finding the decisive error or reaching the reasoning limit.The method maintains a bounded priority queue of variables and explores earlier operations first.
  • Starting-point selection: Hybrid retrieval initializes exploration with question-relevant source messages using dense and sparse retrieval, Reciprocal Rank Fusion, and the top ⌊N/2⌋ fused candidates.The retrieval query concatenates the failed question with the golden answer, while remaining queue capacity is reserved for downstream variables.
  • Graph exploration: At each step, MemTrace selects the earliest-timestamp variable, retrieves its directly involving operations, and presents each operation subgraph as a localized textual representation.The representation includes operation metadata, inputs, outputs, and dependency relations, allowing reasoning over the relevant graph region instead of the entire graph.
  • MemTrace-OBS: MemTrace-OBS addresses inefficient traversal in weakly structured or chain-like graphs by converting operation subgraphs into searchable textual operation blocks.It removes dependency edges and unique identifiers while preserving variables and operation attributes, then supports regular-expression search over the resulting operation log.

5 Experiments

Experiments on MemTraceBench evaluate failure attribution across memory systems using ETA, OIA, token cost, and runtime. Results show that graph-based MemTrace improves attribution, search-based MemTrace-OBS lowers cost, and error patterns expose system-specific bottlenecks and annotation challenges.

  • Attribution accuracy: 18.75% overall OIA is achieved by All-at-Once on GPT-5.4, substantially below MemTrace and MemTrace-OBS because flattened-log analysis discards early operations.This favors retrieval and response errors near the end of execution, making performance depend on error distributions.
  • Attribution accuracy: MemTrace achieves the best error-type attribution with both GPT-4.1 mini and GPT-5.4 backbones, with especially large gains for the smaller model.Graph-based exploration improves error-type attribution, particularly for GPT-4.1 mini.
  • Attribution accuracy: 20.00% to 36.46% is the improvement in overall ETA from MemTrace-OBS to MemTrace, while global operation search can cause GPT-4.1 mini to misclassify retrieval and response errors as extraction errors.MemTrace-OBS may extract keywords from the golden answer and jump directly to nearby retrieval or response operations.
  • Inference cost: 15.25% of MemTrace’s tokens and 29.42% of its runtime are used by MemTrace-OBS on the long-context subset, while the corresponding Mem0 figures are 76.75% and 40.26%.MemTrace-OBS has the lowest overall inference cost across both backbones, with larger reductions for some systems because trace structure differs.
  • Evaluation reliability: LLM-identified errors are almost always judged correct, but annotation errors remain in all three datasets because of imprecise questions, insufficient evidence, or inconsistencies in references.The LLM judge is overly strict on responses that are essentially correct but overly verbose or insufficiently specific.
  • Failure analysis: RAG has no extraction errors, EverMemOS produces very few, and all systems exhibit response errors, indicating distinct bottlenecks across memory systems.RAG lacks an extraction module, while EverMemOS’s extraction module is described as more robust and generalizable.

6 Application

MemTrace aggregates failures at the operation level to generate reports for Mem0 and EverMemOS, revealing distinct failure patterns. In a closed-loop optimization pipeline, MemTrace-guided prompt optimization improves Mem0 and LoCoMo performance by 7.62 points.

  • Failure analysis: MemTrace automatically summarizes where and how failures arise across long construction, update, retrieval, and response pipelines.The analysis is applied to Mem0 and EverMemOS through operation-level failure aggregation.
  • Failure analysis: Mem0 tends to retain high-level user information while dropping fine-grained details, and updates may reassign timestamps without changing content.These patterns are identified in the generated failure reports.
  • Failure analysis: EverMemOS shows no major extraction errors in the generated reports.The passage contrasts this pattern with the extraction failures identified for Mem0.
  • Automatic prompt optimization: 7.62 points: MemTrace-guided optimization raises the LLM-as-a-judge score from 66.70% to 74.32%, while the no-attribution baseline falls to 44.73%.Without localization, the optimizer cannot determine how upstream extraction and memory-update prompts contribute to the final error, producing noisy updates.

7 Related Work

Recent LLM memory systems dynamically manage memories across sessions, but their complex execution pipelines make failures difficult to localize and attribute. Unlike general LLM-agent diagnosis, long-term-memory failures may originate in earlier sessions and must be distinguished from irrelevant interaction history.

  • Recent LLM memory systems dynamically extract, update, forget, and maintain memories across sessions.
  • Their complex execution pipelines make memory failures difficult to localize and attribute.
  • Long-term-memory failures in stateful agents may originate from earlier sessions and must be distinguished from substantial irrelevant interaction history.

8 Conclusion

This work addresses automatic failure diagnosis in non-parametric memory systems by introducing MemTraceBench and MemTrace. The benchmark supports attribution of memory-system failures to concrete execution-pipeline operations.

  • MemTraceBench is built from publicly available datasets and open-source memory systems to study failures in non-parametric memory systems.
  • MemTrace attributes memory-system failures to concrete operations in the execution pipeline.

Limitations

The work is an initial step toward automatic failure attribution for non-parametric memory systems, leaving benchmark expansion and broader validation open for future study.

  • Benchmark scope: MemTraceBench’s scale and diversity could be expanded beyond its current coverage of representative memory systems and long-horizon memory benchmarks.Future benchmarks could include broader forms of memory, such as task memory.
  • Generality: Applying the graph-based automatic diagnosis approach to other compound systems could further test its generality.

Ethics Statement … D.6 Annotation Interface Design

The paper frames MemTrace as a tracing and attribution framework for stateful non-parametric memory systems, whose failures can originate in earlier memory operations. It constructs and annotates execution graphs across diverse datasets and systems, while addressing annotation usability and ethical safeguards for future real-world deployment.

  • B Detailed Discussion of Prior Failure Attribution Work: MemTrace targets stateful agents with persistent non-parametric memory, where failures may originate in earlier operations rather than only the final response.This setting distinguishes it from task-scoped agent attribution and final-context RAG diagnostics.
  • B Detailed Discussion of Prior Failure Attribution Work: The framework represents memory execution through lightweight source-code instrumentation that records persistent-variable creation, updates, deletion, retrieval, and downstream use.This avoids reconstructing dependencies from unstructured logs and supports multi-session memory-state tracing.
  • C Non-Parametric Memory Systems: Long-context and RAG systems fit the non-parametric-memory formulation, while more advanced systems add LLM-based update or read suboperations that complicate error localization.Long-context appends messages and reads the full state; RAG stores messages externally and retrieves top-K results.
  • D.1 Data Sources and Memory Systems: The benchmark combines LoCoMo, LongMemEval, and RealMem with long-context memory, RAG, Mem0, and EverMemOS under offline or online evaluation protocols.LoCoMo and LongMemEval use completed trajectories, whereas RealMem answers using the memory state available before incorporating the task-query message.
  • D.2 Execution Graph Collection Details: 1,514 distinct errors were collected after filtering unanswerable questions from message-by-message execution-graph runs on 4 LoCoMo, 200 LongMemEval, and 3 RealMem trajectories.The collection uses smartcomment tracing statements and selects operation granularity to make attribution meaningful and reduce annotation ambiguity.
  • D.3 Experimental Setup for Memory Systems: The experiments standardize backbone and evaluation settings across systems, using GPT-4.1 mini for memory construction and response generation and an LLM-as-a-Judge protocol.Dense retrieval uses Qwen3-Embedding-4B, EverMemOS reranking uses Qwen3-Reranker-4B, and the judge is Claude Opus 4.5.
  • D.4 Annotation Process: 160 system-related failure cases, 67 annotation errors, and 3 LLM-as-a-Judge errors resulted from independent multi-annotator review followed by consensus discussion when needed.Annotators used a defined taxonomy and extensive guidelines, with system-related cases reviewed until 40 errors per system were identified.

E Multi-Error Attribution … H Prompt Optimization

The paper extends MemTrace to jointly attribute multiple errors, introduces smartcomment for flexible execution-graph tracing, and uses localized faults to make prompt optimization tractable in long-horizon memory systems.

  • E Multi-Error Attribution: A Decisive Error Set identifies a minimal topological frontier of faulty operations that is causally sufficient for the failure.All selected operations must be faulty, their strictly upstream ancestors functionally correct, and correcting their outputs must remove the failure.
  • E Multi-Error Attribution: MemTrace can extend beyond singleton error sets by pruning operations downstream of each identified fault and searching the remaining graph for additional independent faults.The current implementation targets singleton decisive error sets, but the proposed procedure supports non-singleton cases.
  • F Tracing Toolkit: smartcomment collects execution graphs from existing Python systems by recording developer-specified operations and the variables flowing through them without requiring runtime rewrites.It is designed as a lightweight tracing package for instrumenting existing memory-system implementations.
  • F.1 Comparison with Prior Tracing Frameworks: smartcomment combines explicit instrumentation with graph-based provenance, tracing arbitrary Python variables and dependencies through user-defined serializable representations.Unlike instrumentation systems that record only events or call trees, it captures semantic variable and operation relationships for heterogeneous memory workflows.
  • F.2 Design and Features of smartcomment: Its hierarchical model organizes runs into sessions, operations, variables, and dependencies, while lightweight statements mark boundaries, inputs, outputs, and cross-program variable identity.Contextual metadata can attach labels, comments, model settings, generation hyperparameters, and API errors to traced objects.
  • F.3 Instrumentation Example: Mem0 deletion tracing represents a removed memory unit with a constant deletion marker and links the unit to that marker within the current operation context.The dependency edge is automatically associated with the operation identifier, explicitly preserving the deletion effect in the execution trace.
  • G Additional Experimental Details for Failure Attribution: MemTrace experiments use a 272,000-token working-context threshold, 200 reasoning iterations, Qwen3-Embedding-4B initialization, and N = 16 maximum explored operations.The long-context baseline was excluded because scanning the entire trajectory was prohibitively slow and performed poorly.
  • H Prompt Optimization: Localizing faulty operations on the execution graph makes prompt optimization a small, well-scoped problem instead of reasoning over or replaying an entire causal chain.This addresses long multi-session chains that exceed optimizer context windows or cause unreliable attention, prohibitive replay costs, and signal blockage.

H.2 Experimental Details … J.2 Retrieval Query Construction

The experiments optimize Mem0 prompts through developer-guided, operation-level MemTrace attribution and evaluate retrieval-based source-evidence initialization under multiple query constructions. Additional analyses detail reporting, optimization cost, retrieval performance, and deployment-compatible retrieval alternatives.

  • H.2 Experimental Details: Mem0 optimization targets fact extraction, memory updates, and question answering, using three LoCoMo trajectories for optimization and seven for testing across three rounds.The setup assumes a developer understands the target memory system and has data for iterative evaluation.
  • H.2 Experimental Details: MemTrace assigns credit to failed cases, localizes faulty operations, and restricts optimization to prompts participating in each localized operation.The optimizer uses source evidence and pipeline knowledge to explore operation subgraphs before targeting the corresponding prompts.
  • H.2 Experimental Details: The lightweight optimizer aggregates failure feedback across cases, rewrites the target prompt, retains one previous prompt version, and updates Mem0 after each round.The no-attribution baseline uses the same data, backbone, optimizer, and three-round budget but jointly updates all three prompts from the final response-stage subgraph.
  • H.2 Experimental Details: 1.33 minutes per failed case is MemTrace’s average wall-clock runtime, while subsequent optimizer stages remain lightweight because they use localized subgraphs and targeted prompts.The optimization pipeline’s overall cost is dominated by MemTrace, but the reported runtime supports offline prompt optimization.
  • I Report Generation Details: GPT-5.4 synthesizes attribution outputs into error-analysis reports, using minibatches of four and iterative report refinement for EverMemOS.Mem0 uses standard MemTrace diagnostics, whereas EverMemOS additionally uses source evidence and prior knowledge; finer-grained subtypes may be identified when necessary.
  • J.1 Retrieval Performance: Hybrid retrieval performs best across datasets for Recall@8 when comparing BM25 sparse retrieval, Qwen-Embedding-4B dense retrieval, and MemTrace’s RRF hybrid retrieval.On LoCoMo and LongMemEval, even the weakest retrieval method exceeds 70%.
  • J.2 Retrieval Query Construction: 62.20% Recall@8 with the system prediction exceeds 59.50% using the query alone, although golden answers achieve the highest recall.The fixed RRF hybrid retriever is evaluated with deployment-compatible queries because golden answers are generally unavailable in production; incorrect predictions can still provide relevant entities or partial facts.
  • J Additional Analysis: Annotator disagreement is analyzed through pairwise rates, with one annotator pair unevaluable because their annotated cases do not overlap.Figure 7 also reports execution-graph log token distributions for each memory system.

J.3 Additional Dataset Analysis

The analysis finds that memory-system traces are too large for direct long-context inspection and that failure-attribution annotation is difficult. Error sources also vary by memory system, with Mem0 and EverMemOS errors concentrated in LoCoMo and RAG and long-context errors concentrated in LongMemEval.

  • Trace Scale: Each memory system produces traces exceeding one million tokens, with advanced systems approaching 107 tokens, making direct end-to-end inspection difficult.Long-context memory further enlarges logs by recording both pre- and post-update context windows for each update.
  • Annotation Difficulty: Annotator pairwise disagreement ranges from 3% to 46%, indicating that failure-attribution annotation is especially challenging for Mem0 and EverMemOS.
  • Dataset-Source Distributions: LoCoMo contributes the largest share of system-related errors for Mem0 and EverMemOS, whereas LongMemEval contributes most errors for RAG and long-context memory.The passage attributes the latter pattern possibly to LongMemEval’s long user trajectories, which introduce more distracting context.
Loading 2605.28732v3…