Source-linked AI summary
Remember by Asking: Retrieval-Induced Memory Evolution for LLM Agents
Wanqi Zhou, Jiawei Lu, Yang Wang, Zhaolong Xing, Zhen Chen, Ai Han, Haoyue Shi
TL;DR
Direct compression can miss details that matter for unforeseen queries, while full-history fallback is costly. RIME retrieves focused dialogue evidence to form and update memory, then selectively returns to source dialogue when semantic memory is insufficient. On LoCoMo, it leads the compared methods across three quality metrics while using substantially fewer query-time LLM tokens than deliberative memory baselines.
Problem
Direct extraction or compression can overlook useful details needed for unforeseen queries, while relying only on compressed memory prevents their recovery and full-history fallback incurs substantial inference cost.
Source: §1 · p. 1Method
RIME uses generic questions to retrieve dialogue evidence for joint memory formation with relevant historical memories, then selectively retrieves source dialogue when semantic memory is insufficient.
Source: §1 · p. 1, §3.1 · p. 2Results
RIME achieves the best performance across LLM-Judge, F1, and BLEU-1 among compared methods, with substantially fewer query-time LLM tokens than deliberative memory baselines.
Source: §4.2 · p. 4Takeaways & Limitations
Analysis supports question-guided memory formation and selective source retrieval as components of robust and efficient long-term memory.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Long-term memory is essential for language agents to maintain coherent and effective behavior over extended, multi-session interactions. Existing memory systems mainly use retrieval at read time, while write-time memory formation still relies on direct extraction or compression. However, when future information needs are unknown, compressing an entire interaction in one pass can overlook locally important details that may matter later. To this end, we introduce RIME, a retrieval-induced memory framework that shifts memory construction from monolithic compression toward evidence-centered integration. RIME uses generic self-questions to retrieve focused dialogue evidence and grounds memory formation in both the retrieved evidence and relevant historical memories, which are jointly reconciled into an evolving memory bank with temporal and provenance information. At inference time, compressed memory serves as the primary rather than the sole source of evidence: when it cannot support an answer, RIME retrieves relevant source dialogue together with its local context to recover information omitted during memory formation, without resorting to full-history processing. Extensive experiments on LoCoMo with Qwen3-235B-A22B and GPT-5.6 Sol show that RIME consistently achieves the best performance across all three quality metrics among the compared methods, while requiring substantially fewer query-time LLM tokens.
1. INTRODUCTION
RIME addresses the risk that direct compression can discard details needed for unforeseen queries, while full-history fallback is costly. It retrieves focused dialogue evidence before consolidation and selectively returns to source dialogue at inference, achieving strong quality and efficiency results on LoCoMo.
1. INTRODUCTION
- Direct extraction or compression can overlook useful evidence, and discarded details may matter for unforeseen queries; full-history fallback also incurs substantial inference cost.Compressed-memory-only systems cannot recover information discarded during formation.Source: §1 · p. 1
- RIME uses generic self-questions to retrieve focused dialogue evidence, integrates it with local context and relevant historical memories, and selectively retrieves source dialogue when compressed memory is insufficient.The paper reports consistent improvements in memory effectiveness and inference efficiency over existing memory systems.Source: §1 · p. 1
- RIME achieves the strongest performance across LLM-Judge, F1, and BLEU-1 among compared methods, with substantially fewer query-time LLM tokens than deliberative memory baselines.Experiments use Qwen3-235B-A22B and GPT-5.6 Sol.Source: §1 · p. 1
2. RELATED WORK
Prior work develops long-term memory stores, memory construction and maintenance methods, and richer retrieval and reasoning mechanisms. RIME moves retrieval into memory writing and uses focused source-turn retrieval rather than broad history deliberation when compact memory is insufficient.
2. RELATED WORK
- Prior systems store persistent memories [3], consolidate facts or organize evolving notes, selectively retain information, or use structured compression and memory management [11].Other work explores retrieval strategies and reflection, retrieval feedback for memory evolution [14] [15], and full-history fallback [7].Source: §2 · p. 2
3. METHODOLOGY
RIME forms memories by retrieving question-relevant dialogue evidence, expanding it with local context, and reconciling it with historical memories. At inference, it answers from semantic memories first and selectively retrieves source dialogue when that answer is insufficient.
3.1. Retrieval-based Memory Formation
- For each formation question, RIME retrieves the most relevant dialogue turns by cosine similarity, merges recalled turns across questions, and expands them with neighboring context.The retrieved turns are ordered by similarity; two rounds of local context expansion address dependencies between nearby turns.Source: §3.1 · p. 2
- Formation questions act as retrieval cues, independently surfacing relevant dialogue turns before memory construction rather than serving as summarization instructions.The implementation uses six shared questions covering people and relationships, events, activities, plans, states and preferences, and specific factual details.Source: §3.1 · p. 2
- RIME retrieves historical memories using each formation question and its recalled evidence, then jointly extracts facts, deduplicates, resolves references, and reconciles historical information.The operations add, update, delete, or leave unchanged a memory; memories retain temporal information and dialogue provenance.Source: §3.1 · p. 2
- Figure 1 contrasts retrieve-then-consolidate memory formation with direct compression and shows selective source-dialogue retrieval after an insufficient memory-based answer.Formation combines focused evidence, local context, and relevant historical memories.Source: Figure 1 · p. 3
3.2. Selective Memory Use
- At inference, RIME first answers a user query from retrieved semantic memories and retrieves source dialogue when the initial memory-based answer is NONE.Retrieved source turns are expanded with surrounding turns from the same session before producing the final prediction.Source: §3.2 · p. 3
4. EXPERIMENTS
Experiments on LoCoMo compare RIME with long-term memory baselines under two LLM backbones. RIME leads on quality metrics, and ablations support the value of question-guided formation and selective source retrieval.
4.1. Experimental Setup
- Evaluation uses 1,540 non-adversarial LoCoMo questions across single-hop, multi-hop, temporal, and open-domain categories, with Qwen3-235B-A22B and GPT-5.6 Sol.RIME uses six formation queries, retrieving the top 10 dialogue turns and top 5 historical memories per query; at inference it retrieves top 30 semantic memories and, if the initial answer is NONE, top 20 source turns with 15-turn context windows.Source: §4.1 · pp. 3–4
4.2. Main Results
- RIME achieves the best Judge, F1, and BLEU-1 performance with both Qwen3-235B-A22B and GPT-5.6 Sol, while using substantially fewer query-time LLM tokens than deliberative baselines.Table 1 covers 1,540 non-adversarial LoCoMo questions; D-Mem results use Qwen3-235B-Instruct.Source: Table 1 · p. 4, §4.2 · p. 4
4.3. Ablation and Analysis
- Disabling selective source retrieval lowers Judge by 10.59 points on Qwen3-235B-A22B and 3.32 points on GPT-5.6 Sol.Source retrieval triggers more often with Qwen than GPT-5.6 (16.8% vs. 6.0%), consistent with the larger degradation.Source: Table 3 · p. 4, §4.3 · p. 4
- Adding six formation dimensions to full-session compression yields no consistent gain over direct compression, whereas retrieval-cued RIME improves all three metrics.The compared formation strategies use the same selective memory-use procedure.Source: §4.3 · p. 4, Table 2 · p. 4
5. CONCLUSION
RIME forms long-term dialogue memories by retrieving evidence with generic questions before consolidation and selectively revisiting source dialogue when compressed memory is insufficient. On LoCoMo, it leads the compared methods across three quality metrics while using fewer query-time LLM tokens than deliberative baselines.
- RIME achieves the best LLM-Judge, F1, and BLEU-1 performance among compared methods while using substantially fewer query-time LLM tokens than deliberative baselines.The reported experiments use Qwen3-235B-A22B and GPT-5.6 Sol on LoCoMo.Source: §5 · p. 4
- Generic questions retrieve relevant evidence before memory consolidation, and selective source-dialogue retrieval supplements compressed memory when it is insufficient.Source: §5 · p. 4
- Analysis supports both question-guided memory formation and selective source retrieval as effective components for robust and efficient long-term memory.Source: §5 · p. 4