Source-linked AI summary
A Dataset for Document Grounded Conversations
Kangyan Zhou, Shrimai Prabhumoye, Alan W Black
TL;DR
Existing dialogue datasets do not adequately combine task-related information flow, casual conversation, document grounding, and many on-topic turns. This paper introduces a crowd-sourced dataset of human conversations grounded in movie documents and evaluates models with and without document information, finding improvements in engagement, fluency, and perplexity when document sections are used.
Problem
Existing dialogue datasets provide limited document grounding and often lack conversations with many on-topic turns, despite the need to interleave task-related information flow with casual chat.
Method
The paper constructs a crowd-sourced dataset of human conversations about 30 Wikipedia movie documents and compares generative models that use or omit document-section information.
Results
The document-informed SEQS model outperforms the document-free SEQ model on engagement, fluency, and perplexity evaluations.
Takeaways & Limitations
Document grounding provides a source of information that improves the generation of more engaging and fluent conversation responses in this dataset.
Abstract
from arXiv · showhide
This paper introduces a document grounded dataset for text conversations. We define "Document Grounded Conversations" as conversations that are about the contents of a specified document. In this dataset the specified documents were Wikipedia articles about popular movies. The dataset contains 4112 conversations with an average of 21.43 turns per conversation. This positions this dataset to not only provide a relevant chat history while generating responses but also provide a source of information that the models could use. We describe two neural architectures that provide benchmark performance on the task of generating the next response. We also evaluate our models for engagement and fluency, and find that the information from the document helps in generating more engaging and fluent responses.
1 Introduction
Existing dialogue datasets largely separate task-oriented and casual conversation, while many lack document grounding and sustained on-topic exchanges. This paper introduces a movie-based document-grounded dataset and reports that document information improves generated-response quality.
- The dataset targets conversations that interleave task-related information flow with casual chat, addressing the need for data supporting both objectives.
- Many dialogue datasets lack grounding in the information presented during conversations and do not include large numbers of on-topic turns.
- The authors introduce real human conversations grounded in Wikipedia movie articles to improve response grounding, context, and coherence.
- +7.5% preference and +0.96 MOS were achieved by responses generated with document information, while perplexity improved by 11.69 points.
2 The Document Grounded Dataset
The dataset uses crowd-sourced conversations about structured Wikipedia movie documents, collected under two document-access scenarios and organized by document sections. It contains 4112 conversations averaging 21.43 turns, with document-grounding quality measured using BLEU-based ratings.
- Dataset creation: The dataset uses two workers chatting for more than 12 turns about movie-related documents collected through Amazon Mechanical Turk.Movies were selected because they are engaging and widely known, helping participants stay on task.
- Document set creation: Thirty Wikipedia movie documents are divided into four sections containing basic information and three key movie scenes.The basic-information section includes items such as year, genre, director, introduction, and review ratings.
- Conversation scenarios: Conversations are collected in two scenarios: only one worker has the document, or both workers have access to the same document.Both scenarios require the workers to chat for at least 12 turns; instructions differ according to document access.
- Conversation workflow: The workflow reveals document sections progressively, beginning with basic information and showing the next section after the required turns.Users are encouraged, but not required, to discuss information from the newly revealed section.
- Dataset statistics: 4112 conversations average 21.43 turns, including 2128 in scenario 1 and 1984 in scenario 2.A turn is defined as an exchange between two workers.
- Dataset statistics: Conversation turns are mapped to document sections, and conversation quality is assessed by adherence to document information.BLEU measures overlap between conversation utterances and the corresponding document, supporting three quality ratings.
- Dataset statistics: 0.22 versus 0.03 was the average BLEU score for workers with versus without document access.The authors report that document-access workers used document information more, without using it verbatim extensively.
3 Models
The paper compares a baseline sequence-to-sequence model using the current utterance with a section-aware variant that conditions generation on both the utterance and its document section.
- Without section: The baseline SEQ generates a response from the current utterance using a sequence-to-sequence model with attention and a copy mechanism.The copy mechanism guides generation to replace unknown tokens.
- With section: The section-aware SEQS model encodes the current utterance and its corresponding document section before generating the response.It uses the same encoder for both inputs and incorporates the section representation into generation.
- With section: At each decoding step, SEQS combines the previous token embedding with the encoded section representation as its input.The model defines this input as ht = [xt−1; hs].
- Experimental Setup: Both models use a two-layer bidirectional LSTM encoder and an LSTM decoder with 300 hidden units and 0.3 output dropout.Training uses Adam with learning rate 0.001 until validation perplexity converges.
4 Evaluation
The evaluation examines document use, perplexity, engagement, and fluency, comparing models with and without document-section information. Across automatic and human evaluations, section information is associated with better generation quality.
- Evaluation design: The evaluation combines automatic measures with Amazon Mechanical Turk judgments of engagement and fluency.The paper uses perplexity for automatic fluency evaluation and pairwise comparison for engagement.
- Dataset analysis: The dataset analysis measures whether conversation utterances use information from the corresponding document sections.The analysis compares tokens in utterances and sections while excluding recent-context and stop-word overlaps.
- Perplexity: 21.8 versus 10.11 perplexity: SEQS achieves lower test perplexity than SEQ when document sections are included.The paper interprets this as evidence that sections help the generation process.
- Engagement: 43.9% versus 36.4% preference: human evaluators choose SEQS responses more often than SEQ responses for engagement.The “No Preference” option receives 19.6% of selections.
- Fluency: 3.84 versus 2.88 fluency score: SEQS responses receive higher human ratings than SEQ responses on the 1-to-4 scale.Each response was annotated by three unique workers.
5 Conclusion
The paper introduces a crowd-sourced conversation dataset grounded in predefined documents and evaluates the value of document information for response generation.
- 5 Conclusion: The dataset consists of crowd-sourced conversations grounded in a predefined set of documents and is made available for download.The paper uses automatic analyses and human judgments to assess the information’s value.
- 5 Conclusion: SEQS, which uses document-section information, outperforms SEQ on engagement, fluency, and perplexity.This conclusion summarizes the paper’s automatic and human evaluation results.
6.1 Movie lists
The movie list includes six titles spanning superhero, comedy, animation, drama, fantasy, and biographical crime-film contexts.
- Movie list: Batman v Superman: Dawn of Justice is included in the movie list.
- Movie list: Catch me if you can is included in the movie list.
- Movie list: How to Train Your Dragon (2010) is included in the movie list.
- Movie list: The Great Gatsby (2013) is included in the movie list.
- Movie list: The Shape of Water is included in the movie list.
- Movie list: The Wolf of Wall Street is included in the movie list.
6.2 Instructions given to the workers
Workers followed distinct document-access scenarios, with staged movie-document discussions and explicit restrictions to information contained in the documents.
- 6.2.1 Scenario 1: users with document: The document-access instructions required workers to read the document before conversing.
- 6.2.1 Scenario 1: users with document: Document holders were instructed to persuade their partner about watching the movie using document information.
- 6.2.1 Scenario 1: users with document: Workers were encouraged to discuss a new paragraph whenever the document changed.
- 6.2.1 Scenario 1: users with document: Each document contained four short paragraphs, and workers discussed each document for 3 turns before the next one appeared.
- 6.2.1 Scenario 1: users with document: Workers could use document information but could not use personally known information absent from the document.
- 6.2.2 Scenario 1: users without document: Workers without the document asked for the movie name and gathered information before deciding whether to watch it.
- 6.2.3 Scenario 2: both users with document: Scenario 2 gave both workers access to the same document and asked them to discuss its content and express whether they liked the movie.
6.3 Post conversation survey questions
After the conversations, workers rated document understanding, partner responsiveness, conversational smoothness, and whether the interaction changed their viewing intention.
- Survey questions: Workers evaluated whether the document was understandable, whether the other user responded actively, and whether the conversation went smoothly.
- Prior familiarity: Workers reported whether they had watched the movie before.
- Viewing intention: Workers indicated whether they would watch the movie after the other user’s introduction.
6.5 Conversation Example 2
The example conversations show workers using movie-document information to discuss plot, cast, genre, reviews, and interpretations across successive document sections.
- Section 1: The opening exchange identifies Inception and introduces its premise, science-fiction genre, Leonardo DiCaprio, and an 86% Rotten Tomatoes score.
- Section 2: Workers then discuss the director, release year, related films, and unavailable information about awards and box-office earnings.
- Section 3: The conversation expands to supporting actors and character roles, including Tom Hardy as Eames and Ellen Page as Ariadne.
- Section 3: Workers describe the film as confusing, suspenseful, mysterious, and original while discussing its uncertainty about reality and dreams.
- Additional example: A separate Shape of Water exchange discusses Del Toro’s fantasy style, expressive acting, Cold War associations, and magical realism.