PaperlayerView exact revision on arXiv

Source-linked AI summary

Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversations

Nuo Chen, Hongguang Li, Juhua Huang, Baoyuan Wang, Jia Li

arXiv:2402.11975v2cs.CL

TL;DR

Retrieval-based long-term dialogue systems face challenges with retrieval accuracy and growing memory databases, particularly in real-world conversations. COMEDY uses one language model to generate and compress memory and produce responses, with compressive memories of past events, user profiles, and relationship changes. The authors report that COMEDY-GPT4 outperforms context-only and retrieval-based approaches across most metrics.

  • Problem

    Retrieval-based dialogue systems face unpredictable retrieval and increasingly difficult memory-database management as conversations accumulate, while real-world conversational variation further complicates retrieval.

    Source: §1 · p. 1

  • Method

    COMEDY uses one language model to generate session memories, compress past events, user profiles, and relationship changes into a memory, and generate memory-grounded responses.

    Source: §1 · p. 2

  • Results

    COMEDY-GPT4 outperforms context-only and retrieval-based approaches across most metrics and has the best average performance across five evaluating perspectives.

    Source: §3.4 · p. 7

  • Takeaways & Limitations

    The authors report that a singular model can handle long-term conversation interactions with greater consistency and predictability, reduced computational overhead, and no data transfer between multiple models.

    Source: §1 · p. 3

    The authors state that dialogue-system performance remains limited and that understanding the nature of real-world conversations remains a long-standing challenge.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Existing retrieval-based methods have made significant strides in maintaining long-term conversations. However, these approaches face challenges in memory database management and accurate memory retrieval, hindering their efficacy in dynamic, real-world interactions. This study introduces a novel framework, COmpressive Memory-Enhanced Dialogue sYstems (COMEDY), which eschews traditional retrieval modules and memory databases. Instead, COMEDY adopts a "One-for-All" approach, utilizing a single language model to manage memory generation, compression, and response generation. Central to this framework is the concept of compressive memory, which intergrates session-specific summaries, user-bot dynamics, and past events into a concise memory format. To support COMEDY, we curated a large-scale Chinese instruction-tuning dataset, Dolphin, derived from real user-chatbot interactions. Comparative evaluations demonstrate COMEDY's superiority over traditional retrieval-based methods in producing more nuanced and human-like conversational experiences. Our codes are available at https://github.com/nuochenpku/COMEDY.

1 Introduction

Long-term dialogue requires systems to retain and integrate past snippets, context, and preferences, but retrieval-based approaches face retrieval and database-management challenges. COMEDY replaces retrieval with a single-model compressive-memory workflow and is supported by Dolphin, a dataset from real user-chatbot interactions.

1 Introduction

  • Retrieval-based systems typically summarize or store past conversations, manage a memory database, retrieve relevant memories, and pass them with the current conversation to a response generator.The pipeline may use separate modules for memory generation, management, retrieval, and response generation.Source: §1 · p. 1
  • Retrieval-based methods face unpredictable performance when modules work together and retrieved memories may miss conversational nuance; their databases also grow harder to manage as conversations accumulate.Outdated or irrelevant stored information can lead to inaccurate or inappropriate responses.Source: §1 · pp. 1–2
  • Real-world dialogue adds colloquial language, nuanced expressions, shifting topics, and memories spanning multiple chatbot-users, complicating accurate retrieval and database maintenance.The cited contrast is with more structured or predictable dialogues used in some training corpora.Source: §1 · p. 2
  • COMEDY uses a single model to distill session-specific memories, compress them into events, a user profile, and user-bot relationship changes, then generate memory-grounded responses.Its compressive memory is intended to prioritize salient information across past interactions.Source: §1 · p. 2
  • Dolphin provides 100k instruction-tuning samples from real online user-chatbot interactions across session-level summarization, memory compression, and memory-grounded response generation.The dataset is designed to support each critical phase of COMEDY’s operation.Source: §1 · pp. 2–3

2 Methodology

The methodology formulates long-term dialogue as three linked tasks, builds Dolphin from real user-chatbot conversations, and trains COMEDY to extract, compress, and use memory in responses.

2.1 Problem Formulation

  • COMEDY models three linked tasks: extracting session-level memories, compressing them into user portraits and event records, and generating the next response from that memory and current context.An episode is a sequence of prior sessions between one user and chatbot; the current session contains alternating chatbot and user utterances.Source: §2.1 · p. 3

2.2 Task and Datasets Collection

  • Dolphin uses episodes with at least 15 sessions, GPT4-Turbo generation, and annotator review to create memory and response data from user-chatbot dialogues.The source platform lets users define AI characters with distinct personalities and backgrounds; personal identifiers are removed.Source: §2.2 · pp. 3–4
  • 102,882 samples span Tasks 1–3, with 39,999 for memory extraction, 30,695 for compression, and 31,131 dialogue sessions for response generation.The dataset includes data from 3,998 AI characters; the first 15 sessions form memory data and subsequent 1–5 sessions test memory integration in dialogue.Source: §2.2 · pp. 4–5

2.3 COMEDY

  • Mixed-task supervised fine-tuning trains COMEDY on memory summarization, compression, and memory-grounded response generation.The paper states that Task 1 data was much larger and that task volumes were aligned to balance training.Source: Table 1 · p. 4, §2.3 · p. 5
  • Task 3 uses DPO to align responses with memory, using automatically generated preferred responses consistent with memory and dispreferred responses that oppose it.The objective conditions on the compressive memory concatenated with the current dialogue context.Source: §2.3 · p. 5

3 Experiments

The experiments compare COMEDY with retrieval, context-only, and memory-related baselines using automatic metrics for memory tasks and human assessments of generated dialogue. The reported results favor compressive-memory methods across most human-evaluation metrics, while a case study illustrates retrieval difficulties in loosely structured conversation.

3.1 Experimental Setup

  • Experiments use Chinese LLaMA 2 7B/13B backbones and compare COMEDY with retrieval-based, context-only, and memory-related baselines.The retrieval baseline uses COMEDY-13B-generated memories, Text2vec Chinese embeddings, and FAISS; the context-only baseline concatenates prior conversations up to model limits.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 Evaluation Metrics

  • Automatic metrics assess Tasks 1&2, while human scoring and ranking assess Task 3 across five dialogue dimensions.Human scores range from 0 to 3, and rankings can assign tied ranks.Source: §3.3 · pp. 6–7

3.4 Main Results

  • COMEDY performs relatively well on automatic metrics in Tasks 1&2, indicating persona and event recognition and session-memory compression; DPO further improves memorability, consistency, and humanness.The authors report that these memory-task capabilities support more coherent memory-grounded responses in Task 3.Source: §3.4 · p. 7
  • COMEDY-GPT4 outperforms context-only and retrieval approaches across most metrics, leads in Coherence and Engagingness, and has the best five-perspective average; SFT COMEDY surpasses ChatGPT, while COMEDY-13B DPO is comparable to GPT4.The human scoring and ranking results cover memory-grounded response generation in Task 3.Source: Table 3 · p. 6, Table 4 · p. 6, §3.4 · p. 7
  • All evaluated models average below 2 in real-world long-term conversations, indicating substantial room for improvement despite COMEDY’s relative gains.Source: §3.4 · p. 7

3.5 Case Study

  • In a loosely structured, aimless chat, COMEDY-13B DPO uses accumulated user information to produce a personalized reply, whereas retrieval-based methods may respond generally.The example response refers to the user’s favorite roasted chicken wings.Source: §3.5 · pp. 7–8

3.6 Discussion

  • The discussion examines mix-training versus solo training for Task 3 and automatic versus random selection of dispreferred DPO samples.The second comparison is discussed in Appendix F.Source: §3.6 · p. 8

4 Conclusion

COMEDY replaces retrieval modules with a single-LLM framework for session-memory extraction, compression, and memory-grounded dialogue generation, supported by Dolphin, a Chinese dataset of more than 100k training samples from genuine user-chatbot dialogues.

4 Conclusion

  • COMEDY eschews a standard retrieval module, and Dolphin supplies more than 100k training samples across three tasks from genuine online user-chatbot dialogues.The paper reports that COMEDY generates more coherent and contextually appropriate memory-grounded responses than other baselines.Source: §4 · p. 8

Limitations

The authors describe current dialogue systems as still limited overall, with understanding real-world conversations remaining a long-standing challenge; they call for further optimization strategies for memorability and engagingness.

Limitations

  • Current dialogue systems remain limited overall, and understanding the nature of real-world conversations remains a long-standing challenge.The authors say COMEDY and Dolphin could contribute to more coherent memory-grounded responses, but do not resolve this broader challenge.Source: Limitations · p. 9
  • Further optimization strategies are needed to maintain memorability and engagingness.Source: Limitations · p. 9

Ethical Concerns

The dataset's ethical concerns include protecting privacy and user trust, minimizing bias, and respecting the intellectual property and intentions associated with user-defined AI characters.

Ethical Concerns

  • Dolphin development prioritizes user privacy and ethical standards to comply with legal requirements and maintain user trust and system integrity.Source: Ethical Concerns · p. 9
  • The dataset aims to minimize bias through balanced representation of diverse dialogues and scenarios, with regular audits to identify and address potential bias or ethical issues.Source: Ethical Concerns · p. 9
  • User-defined AI characters are handled in line with users’ intentions and ethical standards, with care taken to avoid misuse or misrepresentation.Source: Ethical Concerns · p. 9

A Related Works

The section situates long-term conversation as a key challenge for open-domain dialogue and describes retrieval-based memory systems and their management difficulties. It presents COMEDY and Dolphin as an alternative framework and training resource.

A Related Works

  • Open-domain dialogue systems span applications from customer service to personal assistants, while retaining key history information is central to long-term conversation.Source: A Related Works · p. 11
  • Retrieval-based systems summarize or store past dialogue, manage a memory database, and retrieve memories relevant to the current conversation.Xu (2022) describes summarizing past sessions into stored memories, while Bae et al. (2022) describe database management operations.Source: A Related Works · p. 11
  • COMEDY avoids a retrieval module and huge database, while Dolphin provides real-world long-term conversation data to support its training.Source: A Related Works · p. 11

B Filtering Toxic and Useless information

Dolphin filtering combines an initial harmful-content screen, keyword and content-validity checks, and a minimum dialogue-length rule. The authors say these steps support dataset cleanliness and safety.

B Filtering Toxic and Useless information

C Human Evaluation Scheme

Human annotators assessed chatbot interaction quality using a 0–3 agreement scale. The scheme evaluates engagement, human-likeness, memory, coherence, and consistency.

C Human Evaluation Scheme

  • Coherence ratings range from frequently off-topic or irrelevant responses to consistent understanding and perfect coherence.The scale uses four ordered statements, from 0 to 3.Source: C Human Evaluation Scheme · p. 12
  • Consistency ratings range from erratic or contradictory responses to complete consistency throughout the conversation.The listed statements include ratings 0, 1, 2, and 3.Source: C Human Evaluation Scheme · p. 12
  • Engagement ratings range from no desire to continue chatting to full engagement and willingness to continue.The scale uses four ordered statements, from 0 to 3.Source: C Human Evaluation Scheme · p. 12
  • Human-likeness ratings range from robotic responses to responses described as indistinguishable from a human’s.The scale uses four ordered statements, from 0 to 3.Source: C Human Evaluation Scheme · p. 12
  • Memory ratings range from recalling no earlier details to remembering everything with proper proactive responses.The supplied scale statements cover ratings 0, 1, 2, and 3.Source: C Human Evaluation Scheme · p. 12

D Correlation of Human Annotator

Annotator scores show strong positive correlation, with coefficients nearing 1.0. The accompanying appendix materials describe prompts for DPO sample selection, Dolphin annotation, and COMEDY training.

D Correlation of Human Annotator

E Prompts

  • Figure 4 is identified as an overview of the COMEDY training pipeline.Source: Figure 4 · p. 16
  • The prompt materials cover DPO sample collection, GPT4-Turbo Dolphin annotation, and prompts for training COMEDY.These are identified as Tables 6, 7, and 8, respectively.Source: Table 6 · p. 15, E Prompts · p. 16

F Ours VS. Random sampling for depreferred Sample

The proposed automatic strategy for selecting DPO samples performs better than random sentence sampling, especially on memorability and humanness. The random baseline uses utterances from previous sessions in the same episode as dispreferred samples.

F Ours VS. Random sampling for depreferred Sample

Loading 2402.11975v2…