Source-linked AI summary
On Memory Construction and Retrieval for Personalized Conversational Agents
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Jianfeng Gao
TL;DR
Long-term conversational agents need coherent, personalized responses, but common memory granularities limit retrieval accuracy and retrieved-content quality. The paper proposes SECOM, which segments conversations into topical memory units and denoises them with prompt compression; SECOM outperforms baselines on LOCOMO and Long-MT-Bench+.
Problem
Turn-level, session-level, and summarization-based memories have limitations in retrieval accuracy and the semantic quality of retrieved content for long-term conversations.
Method
SECOM constructs a memory bank from topically coherent conversation segments and applies LLMLingua-2 compression-based denoising before retrieval.
Results
SECOM outperforms baseline approaches on the long-term conversation benchmarks LOCOMO and Long-MT-Bench+.
Takeaways & Limitations
Segment-level memory and compression-based denoising contribute to more effective long-term conversational-agent memory retrieval.
Abstract
from arXiv · showhide
To deliver coherent and personalized experiences in long-term conversations, existing approaches typically perform retrieval augmented response generation by constructing memory banks from conversation history at either the turn-level, session-level, or through summarization techniques.In this paper, we present two key findings: (1) The granularity of memory unit matters: turn-level, session-level, and summarization-based methods each exhibit limitations in both memory retrieval accuracy and the semantic quality of the retrieved content. (2) Prompt compression methods, such as LLMLingua-2, can effectively serve as a denoising mechanism, enhancing memory retrieval accuracy across different granularities. Building on these insights, we propose SeCom, a method that constructs the memory bank at segment level by introducing a conversation segmentation model that partitions long-term conversations into topically coherent segments, while applying compression based denoising on memory units to enhance memory retrieval. Experimental results show that SeCom exhibits a significant performance advantage over baselines on long-term conversation benchmarks LOCOMO and Long-MT-Bench+. Additionally, the proposed conversation segmentation method demonstrates superior performance on dialogue segmentation datasets such as DialSeg711, TIAGE, and SuperDialSeg.
1 INTRODUCTION
Long-term, open-domain conversations require agents to retain past events and user preferences, but existing memory granularities have retrieval and context-quality limitations. The paper motivates segment-level memory and compression-based denoising as responses to these challenges.
- Long-term open-domain conversations span multiple sessions and topics, requiring agents to retain past events and user preferences for coherent, personalized responses.
- The paper asks which memory granularity is most effective and whether a novel memory structure can outperform turn-level, session-level, or summarized formats.
- Segment-level memory partitions conversations into topically coherent units, balancing relevant context against irrelevant content while bypassing summarization.
- LLMLingua-2 is used as compression-based denoising because natural-language redundancy can act as retrieval noise.
- Turn-level, session-level, and summarization-based memories each struggle to provide precise retrieval and complete, relevant, coherent context.
- SECOM outperforms baselines on LOCOMO and Long-MT-Bench+, with ablations confirming contributions from segment-level memory and compression-based denoising.
2 SECOM
SECOM constructs memory from topical conversation segments and retrieves compressed memory units. Its segmentation model supports zero-shot operation and can use limited annotations through iterative reflection.
- 2.1 PRELIMINARY: The conversation history consists of sessions, each containing sequential user-agent interaction turns represented as requests and responses.
- 2.1 PRELIMINARY: Memory construction maps conversation history into memory units, while retrieval selects relevant units under a context budget for response generation.
- 2.2 CONVERSATION SEGMENTATION: SECOM splits each session into topical segments and constructs a segment-level memory bank whose units correspond to segments.
- 2.2 CONVERSATION SEGMENTATION: Each segment is defined by contiguous first and last turn indices, with the next segment beginning immediately after the previous one ends.
- 2.2 CONVERSATION SEGMENTATION: GPT-4 serves as the segmentation model because open-domain segmentation lacks large annotated datasets and has ambiguous boundaries.
- 2.2 CONVERSATION SEGMENTATION: With limited annotations, the method selects hard examples using WindowDiff and iteratively updates segmentation guidance through LLM reflection.
- 2.3 COMPRESSION BASED MEMORY DENOISING: Before retrieval, a compression model removes redundancy from memory units; SECOM specifically uses LLMLingua-2 for this denoising function.
3 EXPERIMENTS
The experiments evaluate SECOM against memory-granularity baselines on long-term conversation benchmarks and assess its robustness across retrievers, generators, context budgets, and segmentation settings. Results consistently favor segment-level memory, compression-based denoising, and the proposed segmentation model.
- Main Results: SECOM outperforms all baseline approaches on LOCOMO and Long-MT-Bench+, with a particularly significant advantage on LOCOMO.The main comparison uses 4k-token retrieval context on LOCOMO and 1k-token context on Long-MT-Bench+.
- Retriever Robustness: SECOM is more robust to the deployed retrieval system because topical segments combine coherent relevant information with less irrelevant content.Turn-level units may be fragmented, whereas session-level units contain substantial irrelevant information; both make retrieval sensitive to the retriever.
- Memory Granularity: Segment-level memory consistently outperforms turn-level and session-level memory across varying context budgets in end-to-end question answering.Compression-based denoising was applied in these experiments to isolate the effect of memory granularity.
- Compression Ablation: Removing compression-based memory denoising causes a performance drop of up to 9.46 GPT4Score points on LOCOMO.The ablation attributes the improvement to better memory retrieval.
- Robustness: SECOM remains advantageous with Mistral-7B-Instruct-v0.3 as the response generator, demonstrating generalization across LLM-powered conversation agents.The robustness evaluation uses Mistral-7B-Instruct-v0.3 on Long-MT-Bench+.
- Conversation Segmentation: The segmentation model consistently outperforms baselines in unsupervised evaluation and generalizes well in transfer learning after reflection on 100 challenging source examples.In transfer learning, it surpasses a baseline trained on the full source dataset and some supervised baselines.
4 RELATED WORKS
Prior work highlights the difficulty of managing long-term conversational context and the importance of selecting effective retrieval units. Existing approaches use retrieval, summarization, denoising, and semantic chunking, but document-focused segmentation leaves conversational chunking underexplored.
- Long-term open-domain conversations require memory management to retain background information, extract persona, and understand user intent.
- Retrieval-augmented memory systems select relevant summaries or memory records using dense retrieval or LLM-based retrieval.
- Chunking granularity determines retrieval-unit boundaries, while ineffective or non-coherent segmentation can produce incomplete or noisy units that impair retrieval and response generation.
- Redundant dialogue noise can impair conversational search, motivating denoising approaches based on summaries or prompt compression.
5 CONCLUSION
The paper concludes that memory granularity limits retrieval-augmented response generation and presents SECOM as a segment-level, compression-denoised memory management system. Experiments and ablations support the effectiveness of both design components.
- SECOM constructs memory banks from segment-level units and applies compression-based denoising to improve retrieval for long-term conversational agents.
- SECOM’s experiments demonstrate effectiveness in handling long-term conversations.
- Further analysis and ablation studies confirm contributions from segment-level memory units and compression-based denoising.
A.1 DETAILS OF CONVERSATION SEGMENTATION MODEL
The segmentation model uses GPT-4-0125 in a zero-shot setup to generate all segmentation indices at once and outputs them in JSONL format for downstream processing.
- GPT-4-0125 serves as the backbone language model for conversation segmentation.
- The zero-shot prompt generates all segmentation indices simultaneously rather than iteratively.This design avoids the iterative process used in LumberChunker, which can incur unacceptable latency.
- Segmentation outputs use JSONL format to facilitate subsequent processing.
A.2 ADDITIONAL COST ANALYSIS
The cost analysis compares memory construction, retrieval, and response generation across methods. SECOM improves performance over the baseline with only a slight increase in computational overhead and is more efficient and effective than MemoChat.
- Table 5 compares costs for memory construction, memory retrieval, and response generation across methods.
- SECOM significantly improves performance over the baseline while only slightly increasing computational overhead.
- SECOM outperforms MemoChat in both efficiency and effectiveness.
A.3 THE ANALOGY BETWEEN THE REFLECTION AUGMENTATION AND PREFIX-TUNING
The paper frames reflection-augmented segmentation guidance as analogous to prefix-tuning: both iteratively update an auxiliary guidance representation without changing the underlying model parameters.
- Segmentation prompt: The zero-shot segmentation prompt groups successive exchanges sharing a topic and creates new segments when the topic shifts.The prompt requires JSONL output, with each dictionary representing one segment of one or more exchanges.
- Analogy to prefix-tuning: Reflection augmentation mirrors prefix-tuning by optimizing guidance rather than fine-tuning the language model parameters.Prefix-tuning prepends a learned prefix to Transformer activations, while this method updates textual segmentation guidance.
- Segmentation guidance: The segmentation prompt is treated as a textual guidance G that directs the model toward improved segmentation outcomes.G starts empty and is updated iteratively using model reflection on segmentation mistakes.
- Iterative reflection: The update analogy is expressed as Gm+1 = Gm −η∇L(Gm), with the LLM implicitly estimating the segmentation-loss gradient.The updated guidance is intended to incorporate corrections from the model’s reflection on ground-truth segmentation.
A.4 PROMPT FOR GPT-4 EVALUATION
The evaluation uses both single-response scoring and pairwise comparison, with response order alternated to reduce position bias.
- Evaluation design: LLM-powered evaluation combines single-sample GPT4Score with pair-wise response comparison.The prompts used for evaluation are displayed in Figure 12.
- Evaluation design: Pair-wise comparisons repeat each response pair with reversed order to minimize position bias.
A.5 EVALUATION RESULTS ON THE OFFICIAL QA PAIRS OF LOCOMO
SECOM is evaluated on LOCOMO’s official QA pairs and remains superior on this reproducibility-focused evaluation.
- Official LOCOMO evaluation: SECOM shows superior effectiveness and robustness on LOCOMO’s recently released official question-answer pairs.The evaluation uses the official QA pairs to support reproducibility.
A.6 CASE STUDY
The case studies and additional experiments examine why segment-level memory helps, how retrieval is evaluated, and whether SECOM generalizes across datasets, models, and segmentation settings.
- Case study: Turn-level memory can fragment relevant information, while session-level memory can include multiple topics and substantial irrelevant content.These two failure modes can respectively omit necessary information or distract the response model.
- Case study: Summarization-based methods may omit detailed information needed to answer the user’s request accurately.The comparison includes RecurSum and ConditionMem.
- Retrieval measurement: DCG evaluates retrieval by weighting relevant turns according to their ranked positions, with relevance distributed across consecutive required turns in Long-MT-Bench+.Irrelevant turns receive zero relevance, and this setup supports evaluation across memory granularities.
- Additional benchmarks: SECOM consistently outperforms baselines on CoQA and Persona-Chat, covering QA and next-utterance prediction settings.The experiments target diverse dialogue scenarios, including open-ended and multi-turn interactions.
- Smaller segmentation models: SECOM retains an advantage when GPT-4-Turbo is replaced by Mistral-7B-Instruct-v0.3 or a RoBERTa-based segmentation model.This experiment tests applicability in resource-constrained environments.
- Segmentation evaluation: The segmentation module is evaluated on DialSeg711, TIAGE, and SuperDialSeg in zero-shot and transfer-learning settings.In transfer learning, the model learns a segmentation rubric through reflection on challenging examples.