Source-linked AI summary
Beyond Goldfish Memory: Long-Term Open-Domain Conversation
Jing Xu, Arthur Szlam, Jason Weston
TL;DR
Long-term open-domain conversation is understudied because existing datasets and models focus on short contexts. The paper releases a five-session human-human dataset and studies retrieval and summary-memory architectures. Both outperform conventional encoder-decoder Transformers in automatic and human evaluations.
Problem
Existing open-domain dialogue datasets and models primarily address short, single-session conversations rather than long-term conversational context.
Method
The paper collects Multi-Session Chat and studies retrieval-augmented models alongside a read-write memory model that summarizes and stores conversations.
Results
Both long-context techniques outperform conventional encoder-decoder Transformers in automatic metrics and human evaluations.
Takeaways & Limitations
Multi-Session Chat provides a task and resources for evaluating progress in long-context dialogue modeling.
Takeaways & Limitations
The models raise concerns about toxic language, bias, and other issues during language generation.
Abstract
from arXiv · showhide
Despite recent improvements in open-domain dialogue models, state of the art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state of the art.
1 Introduction
Long-term open-domain conversation remains poorly addressed because standard datasets and models focus on short, single-session exchanges. The paper introduces Multi-Session Chat and evaluates long-context approaches for this setting.
- Existing dialogue datasets typically contain only 2-15 turns from a single conversational session.
- Standard Transformer-based dialogue models cannot incorporate long-term conversational context effectively.
- Multi-Session Chat contains human-human chats across five sessions, with summaries of important personal information for later conversations.
- The paper studies retrieval-augmented models and a read-write memory model that summarizes and stores conversation online.
- Both long-context techniques outperform conventional encoder-decoder Transformers in automatic metrics and human evaluations.
2 Related Work
Prior open-domain dialogue research uses diverse datasets and persona-based context, but conversations are generally short. The paper also connects long-term dialogue modeling with conversation summarization.
- Open-domain dialogue datasets cover many skills, but their conversations are typically short.
- Persona-based datasets provide personal knowledge as a relatively simple form of dialogue memory.
- Most summarization research concerns documents, while some work addresses dialogue summarization.
- Transformer self-attention makes very large contexts inefficient, contributing to short truncation lengths in existing models.
3 Multi-Session Chat
Multi-Session Chat models long-term conversation as repeated short chats that resume after hours or days. Its design combines role-based data collection, session summaries, and evaluation beyond the training-session distribution.
- Dataset design: MSC treats long conversations as successive focused sessions that resume after hours or days and revisit old or new topics.
- Data collection: Crowdworkers play assigned personas rather than discussing their real identities or personal information.
- Data collection: Session 1 uses PersonaChat, while later sessions reuse the same roles after simulated intervals of 1-7 hours or 1-7 days.
- Dataset scale: Training includes 3- and 4-session episodes, while validation and test conversations extend to 5 sessions.
- Conversation summaries: Between sessions, conversations are summarized into important points that serve as shorter references for subsequent dialogue.
- Dataset scale: Training conversations average ∼53 utterances across four sessions, compared with ∼66 utterances across five-session validation and test conversations.
4 Modeling Multi-Session Chat
The paper compares standard long-context Transformers with retrieval augmentation and a memory architecture that summarizes pertinent information before storing it. Retrieval and summary memory reduce the burden of using long conversational histories.
- Encoder-decoder baseline: The baseline is a BST 2.7B encoder-decoder Transformer fine-tuned on MSC, with experiments extending encoder context from 128 to 256, 512, or 1024 tokens.
- Retrieval augmentation: Retrieval augmentation selects relevant portions of the conversation history for the decoder instead of encoding the entire context.
- Retrieval augmentation: The retrieval models store dialogue memories, score them against the current context, and use the top N items for generation.
- Summary memory: The proposed summary memory first summarizes new pertinent information and adds it to long-term memory.
- Summary memory: A memory-augmented generator uses the dialogue context and long-term memory to generate the next response.
5 Experiments
Experiments evaluate dialogue and summary context, session length, retrieval augmentation, summary memory, and human engagement in Multi-Session Chat. Long-context and memory-based methods generally improve automatic and human-evaluation performance, especially at session openings.
- Context and summaries: Previous session dialogue improves perplexity after the first session, with larger context lengths producing greater improvements.
- Context and summaries: Gold summaries improve perplexity over dialogue or no context across later sessions and context lengths, although gains are not large.Summaries can be less frequently truncated because they are shorter.
- Session openings: Session openings show around 2 perplexity points difference between using and omitting previous-session context.Openings reengage speakers using shared-interest information from earlier sessions.
- Summary models: Removing either partner or self summaries harms performance, while retaining both is best; self summaries are slightly more important.These differences are magnified for session-opening performance.
- Training sessions: 1.5 perplexity points are gained with more than one training session versus one, but increasing training sessions from 1–3 to 1–4 yields only 0.03 averaged gain.
- Retrieval and memory: SumMem-MSC 2.7B (FiD-RAG) performs best among tested methods, while all summary-memory models outperform their retrieval-augmented counterparts.The reported ordering is FiD-RAG above FiD above RAG, with FiD and FiD-RAG exceeding truncation at session openings.
- Human evaluation: 62.1% engaging responses and a 3.65 final rating are reported for SumMem-MSC 2.7B (RAG), versus 53.0% and 3.14 for BlenderBot.Partner-topic references also rise to 33.8% from BlenderBot’s 14.5%.
6 Conclusion
The conclusion reports that existing dialogue data and models inadequately support long-term conversation. The proposed task and long-context approaches improve evaluation performance and support further research.
- Existing dialogue approaches fail to conduct long-term conversations adequately in both their training data and trained models.
- Multi-Session Chat provides a crowdsourced task for training and evaluating long-context dialogue models.
- The long-context dialogue modeling approach outperforms previous systems in automatic metrics and human evaluations.
- Future work should investigate further architectural improvements for long-context dialogue.
7 Societal Impact
The societal-impact discussion focuses on safety concerns associated with large language models and clarifies how the study handles personal information during data collection.
- The dialogue models share concerns about toxic language, bias, and other issues during language generation because they use large language models.
- Crowdworkers played assigned personalities rather than themselves and were not identifying personal information during data collection.
A Extra Results
Additional materials provide example conversations, model predictions, evaluation-task screenshots, and a table comparing validation perplexity across model families.
- Figures 3 and 4 show session-opening annotations and predictions given gold summaries on validation data.
- Figure 5 presents a four-session Multi-Session Chat conversation in which later sessions revisit earlier subjects, explore them, or introduce new topics.
- Table 9 compares validation perplexity across retrieval- and memory-augmented models and encoder-decoder baselines.
- Figure 6 shows the crowdworker human-evaluation instructions and conversation panels.