Source-linked AI summary
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long T. Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, Tomas Pfister
TL;DR
Long-term personalized dialogue requires memory that captures semantic conversational structure and adapts retrieval to different contexts. RMM addresses these gaps with prospective topic-based memory organization and retrospective online refinement, and it outperforms strong baselines across retrieval and response-quality evaluations. Its main limitations are computational cost, text-only scope, and the need for more efficient memory updating.
Problem
Fixed memory granularity can fragment dialogue representations, while fixed retrievers struggle to adapt across dialogue domains and user interaction patterns.
Method
RMM organizes dialogue histories into topic-based memories for future retrieval and uses attribution signals to refine retrieval online through reinforcement learning.
Results
RMM outperforms state-of-the-art baselines in retrieval relevance and response quality for personalized dialogue tasks.
Takeaways & Limitations
RMM provides a more nuanced and adaptable memory approach for long-term personalized dialogue.
Takeaways & Limitations
RMM can be computationally expensive, is focused primarily on textual data, and may need further optimization for dynamically evolving long-term interactions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information from long-term interactions limits their effectiveness in applications requiring sustained personalization. External memory mechanisms have been proposed to address this limitation, enabling LLMs to maintain conversational continuity. However, existing approaches struggle with two key challenges. First, rigid memory granularity fails to capture the natural semantic structure of conversations, leading to fragmented and incomplete representations. Second, fixed retrieval mechanisms cannot adapt to diverse dialogue contexts and user interaction patterns. In this work, we propose Reflective Memory Management (RMM), a novel mechanism for long-term dialogue agents, integrating forward- and backward-looking reflections: (1) Prospective Reflection, which dynamically summarizes interactions across granularities-utterances, turns, and sessions-into a personalized memory bank for effective future retrieval, and (2) Retrospective Reflection, which iteratively refines the retrieval in an online reinforcement learning (RL) manner based on LLMs' cited evidence. Experiments show that RMM demonstrates consistent improvement across various metrics and benchmarks. For example, RMM shows more than 10% accuracy improvement over the baseline without memory management on the LongMemEval dataset.
1. Introduction
LLMs need external memory to sustain coherent, personalized dialogue, but fixed memory granularity and retrieval mechanisms limit effectiveness. RMM addresses both issues through prospective topic-based organization and retrospective online retrieval refinement, improving benchmark performance.
- Motivation: LLMs’ statelessness makes it difficult to maintain coherent, personalized conversations over time.Personalization requires recalling relevant information from previous interactions in addition to understanding the immediate context.
- Motivation: External memory mechanisms help dialogue agents maintain personality traits, user preferences, and continuity across interactions.
- Limitations: Fixed turn, session, or interval boundaries can fragment or incompletely represent conversational information when they do not match semantic topic shifts.
- Approach: RMM summarizes dialogue histories into decomposed topics and uses attribution signals from response generation to refine retrieval online without costly labeled data.
- Results: More than 5% improvement over the strongest baseline was observed across memory retrieval and response generation metrics on MSC and LongMemEval.
- Contributions: The paper contributes RMM, experiments on two long-term personalized dialogue benchmarks, and analysis of fixed-granularity and fixed-retriever limitations.
2. Related Work
Prior work spans long-term dialogue evaluation, internal-model methods, summarization-based memory, and memory modules for personalized agents. RMM belongs to summarization-based approaches while specifically targeting fragmented topics and static retrieval.
- Long-term Dialogue: LoCoMo evaluates sustained dialogue and exposes difficulties with long-range temporal and causal understanding.
- Long-term Dialogue: Some existing methods require white-box access to model internals, limiting their use with proprietary or API-based LLMs.
- Memory Methods: Summarization-based methods condense long contexts into structured events or topics for conditioning or retrieval.
- Memory Methods: RMM extends this category by addressing fragmented topics from fixed granularity and adding retrospective retrieval refinement.
- Personalized Agents: Personalized dialogue agents have used rules, heuristics, symbolic representations, and incremental user-profile enrichment to retain past information.
- Personalized Agents: Collecting substantial annotations for training personalized systems for long-term use is difficult.
3. Problem Formulation
The paper formulates personalized dialogue as a multi-session task with external memory. The central challenge is balancing comprehensive storage of salient information with precise retrieval of relevant context.
- Setting: The task involves an agent interacting with a user across multiple distinct sessions, each containing a sequence of user queries and agent responses.
- Setting: An external memory serves as the agent’s sole repository for information beyond the current conversational context.
- Challenges: The agent must proactively identify and store salient information while accurately retrieving relevant information from memory.
- Challenges: Retrieving irrelevant context can distract the LLM and degrade response quality, making precise retrieval essential.
- Challenges: Effective memory management requires balancing comprehensive storage with precise retrieval for personalized and coherent multi-session dialogue.
4. Framework Overview
RMM combines topic-based prospective memory organization with retrospective retrieval refinement. Its pipeline retrieves and reranks memories, generates a response with cited evidence, updates the reranker, and stores new session memories.
- Framework Overview: RMM decomposes dialogue history into topic-based memory representations optimized for future retrieval.
- Framework Overview: The framework uses online feedback signals generated during response generation to dynamically refine retrieval.
- Components: The memory bank stores entries as pairs of topic summaries and raw dialogue, with topic summaries serving as search keys.
- Pipeline: The pipeline retrieves candidate memories with f_θ(q, B) and reranks them with g_ϕ(q, M_K) before response generation.
- Pipeline: When a session ends, ExtractMemory processes the session and UpdateMemory adds the resulting entries to the memory bank.
5. Prospective Reflection: Topic-Based Memory Organization
Prospective Reflection organizes dialogue memory around semantically coherent topics rather than fixed conversational boundaries. After each session, extracted topic memories are merged with related entries or added as new entries to maintain a consolidated memory bank for future retrieval.
- Topic-Based Memory Organization: Prospective Reflection defines a topic as a semantically coherent discussion unit that may span one or multiple turns.
- Memory Extraction: After each session, an LLM extracts dialogue snippets and summaries according to the distinct topics mentioned.
- Memory Update: Extracted memories are compared with semantically similar memory-bank entries before update decisions are made.
- Memory Update: The update process directly adds memories for new topics or merges them with existing memories when they provide updated information.
- Outcome: Prospective Reflection maintains a coherent, consolidated representation of evolving dialogue history organized around meaningful topic structures.
6. Retrospective Reflection: Retrieval Refinement via LLM Attribution
Retrospective Reflection uses a lightweight reranker to refine retrieved memories for diverse dialogue contexts without modifying the underlying retriever. It trains this reranker online with binary LLM citation feedback, aligning future selections with the utility of retrieved evidence.
- Reranker Design: A lightweight reranker refines Top-K retrieved memories and selects Top-M candidates for the query.
- Embedding Adaptation: The reranker adapts query and memory embeddings through linear transformations with residual connections before relevance scoring.
- Stochastic Sampling: Gumbel noise enables stochastic sampling from relevance scores while preserving gradients for differentiable ranking and reinforcement learning.
- Adaptation: The reranker enables retrieval refinement across pretrained retrievers and supports task-specific adaptation through reinforcement learning.
- LLM Attribution: The generator assigns each retrieved memory a +1 reward when cited and a −1 reward otherwise, based on its utility in the response.
- Reranker Update: REINFORCE fine-tunes the reranker using citation-derived rewards, a baseline value, and the gradient of the selected-memory probability.
7. Experimental Setup
The experiments evaluate RMM on MSC and LongMemEval using response-generation and memory-retrieval measures, alongside baselines representing alternative long-term memory strategies. Implementation choices include multiple dense retrievers, configurable retrieval depths, and an oracle condition for LongMemEval.
- Implementation Details: Contriever is the default dense retriever; without a reranker Top-K is 5, while with a reranker the defaults are Top-K 20 and Top-M 5.
- Implementation Details: LongMemEval additionally considers an oracle retriever that retrieves annotated ground-truth turns containing necessary personal knowledge.
- Datasets: Experiments use the MSC and LongMemEval benchmarks for personalized dialogue evaluation.
- Evaluation Metrics: MSC evaluates generated responses with METEOR and BERTScore against human-provided ground truth.
- Evaluation Metrics: LongMemEval uses Recall@K for relevant-memory retrieval and an LLM judge for answer accuracy against human ground truth.
- Baselines: RMM is compared with No History, Long Context, RAG, and personalized dialogue-agent baselines representing different memory strategies.
8. Experimental Results
Across MSC and LongMemEval, RMM consistently outperforms the evaluated baselines, while ablations and analyses identify contributions from its reflective components, reranking, and adaptive retrieval.
- 8.1. Main Results: 0.0% accuracy without history on LongMemEval and 5.2% METEOR on MSC demonstrate the necessity of historical context.Long-context models also remain limited, with MSC METEOR below 20%, BERTScore under 40%, and LongMemEval accuracy below 58%.
- 8.1. Main Results: RMM consistently achieves the best results across datasets and metrics, outperforming Long-Context LLMs, RAG, and personalized dialogue-agent baselines.The comparison covers MSC and LongMemEval using METEOR, BERT Scores, Recall@5, and Accuracy.
- 8.1. Main Results: 86% of MSC responses and 100% of LongMemEval cases improve in response quality when memory is used.The datasets frequently require recalling prior discussion topics or historical information.
- 8.2. Ablation Study: The complete framework achieves 30.8% METEOR on MSC and 60.4% Recall@5 on LongMemEval, while the reranker improves alignment over Retrospective Reflection alone.The ablation reports 27.5% METEOR and 58.8% Recall@5 for the reranker alongside RR.
- 8.3. Granularity Analysis: Adaptive memory organization outperforms fixed turn or session granularity, while mixed granularity underperforms because of increased search noise.Prospective Reflection integrates fragmented conversational segments into cohesive topic-based memories.
9. Conclusion
RMM combines topic-based memory organization with reinforcement-learning-based retrieval reranking. The framework is reported to improve retrieval relevance and response quality while identifying limitations of fixed-granularity and static-retriever approaches.
- RMM integrates Prospective Reflection for structured, topic-based memory organization and Retrospective Reflection for dynamic memory reranking via reinforcement learning.
- RMM outperforms state-of-the-art baselines in retrieval relevance and response quality for personalized dialogue tasks.
- The framework highlights fixed granularity and static retrievers as key challenges in long-term dialogue memory modeling.
Limitations
RMM improves retrieval relevance but has computational, modality, and memory-updating limitations. The authors propose more efficient reranking and broader multimodal support as future directions.
- RMM’s reinforcement-learning-based memory reranking can be computationally expensive for large-scale datasets or real-time applications.
- The current framework primarily focuses on textual data, limiting applicability to dialogue systems incorporating images, audio, or video.
- The memory updating mechanism may require further optimization for dynamically evolving long-term user interactions.
- Future work targets more efficient reinforcement learning, lightweight memory reranking, and extension to multi-modal dialogue systems.
Ethical Statement
The paper notes privacy and fairness considerations for personalized dialogue systems that process historical conversations. It also reports implementation and experimental details, including model, hardware, baseline, and retrieval configurations.
- Ethical considerations: RMM relies on historical conversations that may contain sensitive or private information, motivating encryption and privacy-preserving methods during collection and training.
- Ethical considerations: The authors emphasize transparent data-usage policies, user consent, and efforts to minimize bias in memory retrieval and response generation.
- Experimental setup: The reranker is an MLP with a residual connection, and Retrospective Reflection uses REINFORCE for reinforcement learning.
- Experimental setup: The reinforcement-learning reward is +1 for cited entries and −1 for non-cited entries, with learning rate 1 × 10^-3.
- Experimental setup: Experiments use Gemini-1.5-Flash/-Pro for response generation with a 128k-token context window and temperature 0.0.
- Experimental setup: Experiments use Contriever as the default retriever and integrate MemoryBank and LD-Agent as baselines using the LongMemEval codebase.
- Experimental setup: Figure 6 tracks the ratio of useful memories cited over RL training steps, with the usefulness score increasing from around 0.2 to approximately 0.4 by step 1000.
B. Dataset Description
The paper evaluates long-term personalized dialogue on MSC and LongMemEval, using multi-session histories to test personalized response generation, personal-knowledge question answering, and memory-management behavior. Case studies and extraction examples illustrate how the system tracks evolving preferences and event chronology across sessions.
- MSC: MSC is a multi-session benchmark with turn-level and session-level conversational data annotated for relevance and response quality.Experiments use historical sessions to evaluate human-like personalized responses grounded in prior user preferences and conversation patterns.
- LongMemEval: LongMemEval evaluates questions about specific personal knowledge in extended histories spanning turn, session, and mixed granularities.The experiments sample 100 test instances and use the remaining data for training and validation.
- Case studies: RMM correctly updates a treadmill preference across sessions, whereas the long-context output gives an outdated answer about buying one for home.The case progresses from considering a treadmill to using a gym treadmill because outdoor running is constrained by weather.
- Case studies: RMM correctly identifies the Data Analysis using Python webinar as earlier than the Effective Time Management workshop, while the long-context output remains uncertain.The comparison depends on linking the webinar’s “two months ago” time reference to the event sequence.
- Memory extraction: The memory-extraction examples produce concise personal summaries with turn references, including Speaker_1’s New England weather and fall-season preferences.The extraction task outputs JSON dictionaries containing summaries and reference lists, and returns NO_TRAIT when no personal summary can be extracted.