Source-linked AI summary
Beyond Static Summarization: Proactive Memory Extraction for LLM Agents
Chengyuan Yang, Zequn Sun, Wei Wei, Wei Hu
TL;DR
Existing memory extraction is often ahead-of-time and one-off, which can lose useful details and leave errors or hallucinations uncorrected. ProMem uses multi-grained extraction with completeness checks, atomic fact verification, relation verification, and detail anchoring. Experiments report stronger memory quality and QA performance, while the method's iterative feedback loop increases token consumption and latency.
Problem
Existing methods often extract memory before future tasks are known and only once, risking information loss and persistent extraction errors or hallucinations.
Method
ProMem separately extracts details, events, and relations, recovers omitted events, verifies atomic facts and relations, and anchors details to verified events.
Results
ProMem achieves state-of-the-art performance in memory quality and QA tasks, with Table 1 reporting 81.37% memory integrity while maintaining top accuracy.
Takeaways & Limitations
ProMem provides a high-quality memory foundation that can replace raw dialogue or existing memory while retaining downstream retrieval and management pipelines.
Takeaways & Limitations
The recurrent feedback loop increases token consumption and inference latency, which may bottleneck strictly real-time or resource-constrained applications.
Abstract
from arXiv · showhide
Memory management is vital for LLM agents in long-term and personalized interactions. Most previous work studies how to retrieve and use memory, but pays less attention to how memory is extracted. We find two main limitations in existing methods. First, extraction is "ahead-of-time": the agent saves information before it knows future tasks. A single summary prompt often mixes details, events, and relations, so useful information is lost. Second, extraction is usually one-off. Without verification, errors and hallucinations may stay in memory for a long time. To address these limitations, we propose ProMem, a proactive memory extraction framework. It separates details, events, and relations, and uses different extraction strategies for each type. It also checks completeness to recover missed events and verifies facts at the atomic level to reduce hallucinations. Experiments show that ProMem improves memory completeness and QA accuracy, while keeping a good balance between quality and token cost.
1 Introduction
Existing memory systems often simplify extraction, saving information before future tasks are known and usually extracting only once. ProMem instead extracts memory by type, checks for omissions, verifies facts, and reports stronger memory quality and QA accuracy.
- Summary-based memory methods often emphasize storage, updating, and retrieval while simplifying memory extraction.
- Ahead-of-time extraction can discard small but crucial details because the agent cannot predict future questions.
- One-off extraction can preserve initial mistakes or hallucinations in memory, motivating feedback verification.
- A single-pass extractor can fail to resolve “these activities” as “extreme sports,” producing an ambiguous memory entry.
- ProMem separately extracts details, events, and relations, then recovers missed events and verifies atomic facts against the source dialogue.
- On HaluMem, ProMem is evaluated for constructing more accurate, high-fidelity structured memories and outperforms baselines in memory quality and QA accuracy.The evaluation also examines robustness to token compression and deployment with small language models.
2 Related Work
Related work compresses dialogue or extracts structured information, while iterative feedback and knowledge-graph curation provide relevant ideas. ProMem applies these ideas to personalized multi-turn memory construction, emphasizing completeness and consistency.
- 2.1 Summary-based Memory Management: Summary-based memory systems compress interaction history to support long-horizon tasks under limited context windows.
- 2.1 Summary-based Memory Management: Prior memory research often studies when to summarize and how to organize memory, while treating extraction as passive and static.
- 2.2 Information Extraction: LLM-based information extraction has advanced generative text-to-structure methods, but commonly targets static encyclopedic or news text.
- 2.2 Information Extraction: ProMem extracts personalized user memory from multi-turn dialogue and requires both fact extraction and consistency and completeness checking.
- 2.3 Iterative Feedback and Knowledge Graph Curation: Self-refinement methods review and revise outputs for tasks such as poems or code, whereas ProMem targets memory construction for long-term agents.
- 2.3 Iterative Feedback and Knowledge Graph Curation: Knowledge-graph curation addresses completeness, grounding, validation, entity resolution, and fusion; ProMem applies related operations across memory types and granularities.
3 Proactive Memory Extraction
ProMem replaces one-off memory summarization with an iterative, multi-grained pipeline that recovers omissions, verifies facts, and constructs graph-structured memory. This addresses information loss and hallucinations while trading additional tokens for more complete and accurate memory.
- Motivation: One-off extraction reduces memory from 157K to 12.79K tokens, while QA accuracy drops from 74.13% to 50.66%.The result indicates that a single scan cannot preserve all necessary details.
- Motivation: Increasing extracted memory size brings limited F1 gains, indicating that one-off extraction itself creates a structural bottleneck.
- Overview: ProMem changes static one-pass extraction into an active iterative process that re-checks dialogue, recovers missed details, and removes hallucinations.
- Multi-Grained Memory Disentanglement: Multi-grained extraction separately produces atomic details, higher-level events, and logical relations for a preliminary structured memory graph.Details include times, locations, and entities; events summarize main states, actions, or stories, while relations capture causal or temporal links.
- Explicit Completeness Check: The completeness check compares abstracted events with an initial draft and merges missed events into a more complete memory set.It uses a cross-reference matrix based on event–draft similarity and a completeness threshold to identify events absent from the draft.
- Hallucination Verification and Graph Construction: Hallucination verification matches memory entries to dialogue turns, decomposes problematic entries into atomic facts, and keeps, revises, or discards facts using NLI.The resulting purified memory is supplemented with necessary relation edges and anchored details to form graph-structured memory.
- Relation Verification and Detail Anchoring: The final graph retains purified memory nodes, validated relations, and details anchored to events from the same dialogue turns.ProMem uses necessity-driven relation verification to add supported edges only when flat entries cannot answer a generated probe question.
- Computational Overhead: ProMem uses more tokens than one-pass summarization because event recovery and verification add write-time overhead.The paper argues this trade-off is reasonable because memory errors affect later tasks, extraction is reused, and construction can run in the background.
4 Experiments
Experiments evaluate ProMem across memory quality, downstream QA, token cost, ablations, smaller models, and established benchmarks. Results show strong completeness, accuracy, QA performance, and efficiency, with component-level evidence for completeness and hallucination verification.
- Main results: ProMem achieves the highest memory accuracy and F1 on HaluMem while maintaining 81.37% memory integrity.This balances accurate fact extraction with broad coverage of user information.
- Main results: 11.59K tokens per session gives ProMem a similar budget to Mem0 and better quality than LightMem at comparable cost.At 10.56K tokens, uncompressed LightMem reaches only 55.81% QA accuracy, while ProMem outperforms it across metrics.
- Ablation study: Removing completeness checking lowers integrity from 81.37% to 69.48%, while removing hallucination verification raises hallucination to 21.40%.These ablations support completeness checking for recovering missed events and hallucination verification for removing incorrect facts.
- Ablation study: Removing detail anchoring or relational edges substantially reduces QA, with the w/o Rel variant reaching 61.12% QA accuracy.The results associate fine-grained details with specific-question answering and relations with reasoning.
- Results with SLMs: With Llama3.1-8B, ProMem improves QA accuracy by 10.74% over Mem0 while preserving similar memory accuracy.It also achieves much higher memory integrity, indicating that the framework remains effective with a smaller extraction and QA model.
- Results on LongMemEval and LoCoMo: ProMem reaches 72.12% QA accuracy on LongMemEval-S and outperforms the SOTA LightMem method on downstream evaluation.The comparison frames ProMem’s gains as coming from extraction quality rather than more complex retrieval mechanisms.
5 Conclusion and Future Work
ProMem is presented as a proactive memory-extraction approach, with reported gains in memory quality and downstream QA. Future work extends this direction toward lifelong memory management.
- ProMem achieves SOTA performance in memory quality and QA tasks while supporting efficient deployment using SLMs.
- Table 4 compares Mem0 and ProMem on memory extraction and QA performance.
- Future work will investigate memory updating and forgetting for lifelong memory management.
Limitations
ProMem improves memory extraction and downstream QA but introduces computational and model-capability constraints. Its iterative feedback loop increases cost, and its effectiveness depends on the backbone LLM’s reasoning ability.
- The recurrent feedback loop increases token consumption and inference latency compared with one-pass summarization, potentially limiting strictly real-time or resource-constrained applications.
- ProMem’s self-questioning and verification effectiveness depends heavily on the backbone LLM’s reasoning capabilities.
- The lower bound of model size required for the framework remains an open question despite Llama3.1-8B performing reasonably well.
A Setup for Empirical Study
The empirical study varies extraction granularity on HaluMem to examine traditional memory-management bottlenecks. It evaluates memory completeness, factual correctness, and their harmonic-mean F1 score.
- The study varies extraction intervals from every 2 turns to 4, 6, and 8 turns, extending beyond session-level extraction.
- Memory Integrity (Int.) measures completeness, while Memory Accuracy (Acc.) measures the factual correctness of extracted content.
- F1 is reported as the harmonic mean of Integrity and Accuracy to summarize overall memory quality.
B Component-level Analysis
The component analysis examines detected error types and the costs of ProMem’s verification modules. The results motivate specialized checks, while showing that iterative verification adds measurable write-time cost.
- Detected errors: Hallucination Verification, Completeness Check, and Relation Verification detect unsupported facts, missing events, and missing relations.
- Detected errors: Different extraction errors motivate dedicated verification modules for hallucinations, completeness, and relations.
- Component-level cost: Detail Anchoring has the largest token cost because it links fine-grained details to corresponding events.
- Component-level cost: Completeness Check and Relation Verification add cost for recovering missing events and verifying relations.
- Component-level cost: For GPT-4o-mini, ProMem averages 4.26 LLM calls and 21.38 seconds per session, with costs incurred at write time.
C Results with Token Compression
ProMem remains robust as dialogue information is aggressively compressed, retaining strong memory and QA performance while outperforming Mem0 in low-resource settings.
- At a compression ratio of 0.2, ProMem achieves 37.20% QA and 57.20% memory integrity despite discarding 80% of dialogue tokens.Its performance remains stable as the compression ratio decreases from 0.8 to 0.2.
- At a compression ratio of 0.2, Mem0 falls to 21.34% QA and 23.28% memory integrity.Mem0 is highly sensitive to token reduction in this low-resource setting.
- ProMem maintains superior performance to full-input baselines while reducing token consumption by 60% through aggressive input compression.The authors interpret this result as addressing computational-cost concerns.
D Sensitivity Analysis on τcomp
Increasing τcomp improves extracted-memory completeness through supplementary extraction, while slightly reducing precision and leaving downstream QA performance nearly unchanged.
- Increasing τcomp enhances memory integrity by promoting supplementary extractions.
- Higher τcomp causes a slight decrease in memory accuracy, while downstream QA accuracy fluctuates only marginally.
E Category-level Results on LongMemEval-S and LoCoMo
ProMem's category-level results show broadly consistent QA improvements across question types, outperforming LightMem in most LongMemEval-S categories and all reported LoCoMo categories.
- LongMemEval-S: On LongMemEval-S, ProMem outperforms LightMem in five of six categories and performs similarly on Knowledge-Update.
- LoCoMo: On LoCoMo, ProMem achieves the best performance in all four categories.
- Overall pattern: These results indicate that ProMem's improvement is consistent across different question types.