Beyond Goldfish Memory: Long-Term Open-Domain Conversation

• Common dialogue datasets contain single-session conversations of typically 2–15 turns, and models such as Meena (Adiwardana et al., 2020) and BlenderBot (Roller et al., 2020) truncate context at 128 tokens.These systems are therefore incapable of incorporating long-term conversational context, leaving their performance on multi-session dialogue unclear.Source: §1 · p. 1
• Multi-Session Chat (MSC) provides human-human chats across five sessions, with up to 14 utterances per session and summaries of personal points from earlier sessions.Sessions resume after hours or days, and partners build on known shared interests or explore new ones.Source: §1 · p. 1
• The study examines retrieval-augmented generative models (Shuster et al., 2021) and a proposed read-write memory model that summarizes and stores conversation on the fly.The work also reports extensive experiments and ablations and releases models, data, and code.Source: §1 · pp. 1–2
• Retrieval-augmented generation and a read-write memory model outperform conventional encoder-decoder Transformers, while MSC-trained models show long-term abilities absent from existing state-of-the-art models.The improvements are reported in both automatic metrics and human evaluations.Source: §1 · pp. 1–2

2 Related Work

• Open-domain dialogue datasets cover varied skills but typically span 2–15 turns; crowdsourcing long chats is difficult, and organic online conversations are proprietary and privacy-sensitive.Crowdworkers are typically employed for only a few minutes at a time.Source: §2 · p. 2
• Prior work has used Reddit-extracted personas, crowdsourced personas, and persona information encoded in vector weights (Li et al., 2016).Source: §2 · p. 2
• Document summarization dominates prior work, with some dialogue summarization; this work summarizes long-term conversations to retain useful information (Gliwa et al., 2019; Pan et al., 2018).Source: §2 · p. 2
• Standard Transformers face fixed context limits and inefficient all-vs-all attention; proposed alternatives speed attention or retrieve pertinent context, approaches considered in this work.BlenderBot and Meena use 128-token truncation lengths, while BART uses 1024 tokens; alternatives include long-context Transformers (Child et al., 2019; Beltagy et al., 2020) and retrieval-augmented methods (Shuster et al., 2021).Source: §2 · p. 2

Multi-Session Chat

• MSC treats long-term conversation as short, focused sessions that resume hours or days later, continuing past topics or introducing new ones.The dataset is designed to provide data for both training and evaluating models in this setting.Source: §3 · p. 2

Dataset Statistics

• Four-session training chats average ~53 utterances and five-session validation/test chats ~66, versus 2.6–14.7 utterances in other datasets.The average conversation context is ~1,614 tokens, compared with a 128-token truncation length in the Transformer used by Roller et al. (2020).Source: Table 2 · p. 4, §3 · p. 5

Personas

• Crowdworkers role-play character descriptions rather than discussing themselves, helping mitigate privacy concerns and ensure diversity; 1,155 personas crowdsourced from Zhang et al. (2018) are used, with validation and test personas separate from training.Source: §3 · p. 4

Session 1; Sessions 2, 3, 4, . . .

• Session 1 uses PersonaChat, while later sessions ask workers playing the same roles to catch up and attend to details from prior interactions.Workers simulate either 1–7 hours or 1–7 days between sessions and chat for six turns; workers need not be the same across sessions.Source: §3 · p. 4

Conversation Summaries (Extended Personas); Session Lengths

• Crowdworkers summarize conversations between sessions, and later speakers use these summaries as a concise reference that extends their personas.Training conversations have three or four sessions, while validation and test conversations extend to five sessions to measure performance beyond the training distribution.Source: §3 · pp. 4–5

4.1 Transformer Encoder-Decoders

• The baseline fine-tunes BlenderBot’s BST 2.7B model, extending its 128-token encoder positions to 256, 512, or 1024 while keeping the original 128 unchanged.The added positional embeddings are trained along with the network on the downstream task.Source: §4.1 · p. 6

4.2 Retrieval-Augmentation

• RAG uses DPR to rank FAISS-indexed passages and optimize retrieval end-to-end for generation (Shuster et al., 2021).DPR retrieves from the index and scores the top N candidates.Source: §4.2 · p. 6
• FiD (Izacard and Grave, 2020) separately encodes each retrieved document prepended to the context, then lets the decoder attend to the combined encodings; FiD-RAG uses a RAG-trained retriever (Shuster et al., 2021).The alternative retriever is pre-trained DPR.Source: §4.2 · p. 6
• For retrieval, the authors encode dialogue utterances or whole sessions as memory documents, find whole sessions work better, and select N from 3, 5, or 6 on validation.They store DPR vector encodings for memory items and score them against the dialogue context; the memory is small enough not to require a FAISS index.Source: §4.2 · pp. 6–7

4.3 Summarization Memory-Augmentation

• Summarization memory addresses the volume and unprocessed content of raw history by storing pertinent information extracted from dialogue turns for a memory-augmented generator.An encoder-decoder summarizer can decide that nothing new should be stored; otherwise, its summary is added to long-term memory.Source: §4.3 · p. 7
• Human-annotated MSC summaries supervise the encoder-decoder summarizer, and the retrieval and generation systems from subsection 4.2 can power the memory-augmented response model.The summarizer produces summaries of new pertinent information from the dialogue history.Source: §4.3 · p. 7

Using session dialogue context

• Prior-session dialogue history improves perplexity for every session after the first across all context lengths, with longer histories yielding the best improvement.The authors interpret this as evidence that people use earlier sessions to make later dialogue more salient and that models can use this information when trained on the data.Source: §5 · p. 7

Using summary dialogue context

• Crowdworker-annotated gold session summaries are evaluated as context and may be more informative because they include salient points.Source: §5 · p. 7

5.1 Human Evaluation

• Crowdworkers conversed with a bot as one of two validation-set personas in the fifth session, with summaries of the four prior sessions available.Each conversation had 15 messages, and workers labeled each bot turn for references to persona or discussion knowledge from prior sessions, the current session, or neither, then rated overall engagingness out of 5.Source: §5.1 · p. 10
• SumMem-MSC 2.7B (RAG) achieved a 62.1% engaging response rate and 3.65 final rating, versus BlenderBot’s 53.0% and 3.14 and MSC 2.7B (truncate 1024)’s 54.2% and 3.47.All three summarization-memory variants outperformed encoder-decoders with dialogue histories truncated at 128 or 1024 tokens; MSC-trained models also beat BlenderBot on per-turn engagement and final ratings.Source: §5.1 · p. 10
• All MSC models increased references to partners’ topics from previous sessions, with higher rates for summarization-memory models, while references to their own topics did not particularly increase.SumMem-MSC 2.7B (RAG) referenced partners’ previous-session topics at 33.8%, compared with BlenderBot’s 14.5%; the authors suggest this helps explain its higher engagement ratings.Source: §5.1 · pp. 10–11

• The long-context dialogue modeling approach outperforms previous systems in both automatic metrics and human evaluations.Source: §6 · p. 11
• The work investigates model architectures and collects Multi-Session Chat, a crowdsourced task for training and evaluating long-term dialogue models.Source: §6 · p. 11
• Future work should investigate further architectural improvements for long-context dialogue.Source: §6 · p. 11

• The dialogue models raise concerns about toxic language, bias, and other language-generation issues; Xu et al. (2020) review open-domain mitigation methods.Source: §7 · p. 11
• Crowdworkers played assigned personality roles rather than discussing themselves, while trained models retain exchange-derived information long-term in private conversation memory not shared with others.Source: §7 · p. 11

Session Opening Examples

• Figure 4 shows validation-set session-opening predictions from a model trained on gold summaries alongside the corresponding human annotations.Source: Figure 4 · p. 13, A Extra Results · p. 13

MSC Dataset

• Figure 5 presents a four-session Multi-Session Chat conversation in which later sessions revisit earlier subjects, explore them, or introduce new topics.Source: A Extra Results · p. 13, Figure 5 · p. 14

Main Validation Results

• Table 9 reports validation perplexity across sessions for retrieval- and memory-augmented models and encoder-decoder baselines, corresponding to test perplexity in Table 7.Source: A Extra Results · p. 13, Table 9 · p. 15

Crowdworker Tasks

• Figure 6 shows screenshots of the crowdworker evaluation task, with instructions in the left panel and the conversation in the right panel.Source: A Extra Results · p. 13, Figure 6 · p. 15