Source-linked AI summary
Towards Exploiting Background Knowledge for Building Conversation Systems
Nikita Moghe, Siddhartha Arora, Suman Banerjee, Mitesh M. Khapra
TL;DR
Existing dialog datasets generally omit explicit background knowledge, although human conversation relies on recalling and adapting topic knowledge. The paper introduces a movie-chat dataset with responses linked to background spans and evaluates generation, copy-and-generate, and span-prediction models; span prediction outperforms generation in the oracle setting, while scaling and naturalness remain limitations.
Problem
Existing dialog datasets usually provide utterance sequences without explicit background knowledge, despite human reliance on topic knowledge during conversation.
Method
The paper crowdsources movie conversations whose responses copy or modify content from plots, reviews, comments, and facts, while recording source spans.
Results
Span prediction outperforms generation-based models in the oracle setting.
Takeaways & Limitations
The dataset supports research on models that exploit well-formed background sentences instead of generating every response from scratch.
Takeaways & Limitations
BiDAF requires 256-word resources because longer documents cause out-of-memory errors, and predicted spans can be unnatural without suitable modification.
Abstract
from arXiv · showhide
Existing dialog datasets contain a sequence of utterances and responses without any explicit background knowledge associated with them. This has resulted in the development of models which treat conversation as a sequence-to-sequence generation task i.e, given a sequence of utterances generate the response sequence). This is not only an overly simplistic view of conversation but it is also emphatically different from the way humans converse by heavily relying on their background knowledge about the topic (as opposed to simply relying on the previous sequence of utterances). For example, it is common for humans to (involuntarily) produce utterances which are copied or suitably modified from background articles they have read about the topic. To facilitate the development of such natural conversation models which mimic the human process of conversing, we create a new dataset containing movie chats wherein each response is explicitly generated by copying and/or modifying sentences from unstructured background knowledge such as plots, comments and reviews about the movie. We establish baseline results on this dataset (90K utterances from 9K conversations) using three different models: (i) pure generation based models which ignore the background knowledge (ii) generation based models which learn to copy information from the background knowledge when required and (iii) span prediction based models which predict the appropriate response span in the background knowledge.
1 Introduction
The paper argues that conversation models need explicit background knowledge because human movie conversations draw on plots, reviews, comments, and facts. It introduces a crowdsourced dataset linking responses to such resources and evaluates three background-aware modeling paradigms.
- Existing dialog datasets omit explicit background knowledge, encouraging sequence-to-sequence models that can produce syntactically incorrect or off-topic responses.
- The dataset asks workers to discuss movies using plots, reviews, comments, and fact tables, constructing context-relevant responses from those resources.
- The collection contains around 9K conversations and 90K utterances about approximately 921 movies, with each response linked to its source span.
- The dataset is domain specific but is intended as a benchmark for developing background-knowledge-aware models transferable through similar datasets in other domains.
- The baselines include a background-agnostic hierarchical generator, a copy-and-generate model, and a span-prediction model.
2 Related Work
Prior dialog research includes sequence-generation systems and efforts to integrate external knowledge, but existing resources often lack open, domain-specific conversations explicitly connected to background text. This work instead constructs such a dataset using crowdsourcing and explicit resources.
- Large-scale dialog datasets helped establish end-to-end systems that treat conversation as sequence generation.
- Knowledge integration has been pursued because people rely on background knowledge in conversation and related tasks.
- Existing knowledge-grounded datasets are often templated or use small knowledge graphs rather than open conversations in specific domains.
- Recent open-domain systems have used Reddit, Amazon Evi, OMDB, Google Knowledge Graph, and Wikidata as external sources.
- The proposed collection uses crowdsourcing and explicitly instructs workers to use clean sentences from external knowledge sources, reducing noise through human involvement.
- Unlike self-dialog work, this approach provides explicit background knowledge sources to workers during conversation collection.
3 Dataset
The dataset is built from movie-specific resources and crowdsourced self-chats in which responses are linked to background spans. It covers 921 movies and records resource usage, conversation statistics, and human-verification procedures.
- The dataset covers 921 movies across 22 genres selected from popular movie lists and titles with sufficient online background information.
- Each movie’s background knowledge includes reviews, Wikipedia plots, Reddit comments, and factual metadata such as box office, awards, similar movies, and taglines.
- Workers first received movie-focused opening statements, enabling conversations to begin directly with movie discussion rather than greetings and general chit-chat.
- Crowdsourced self-chats required workers to use provided resources, modify selected spans for coherence, and produce responses explicitly linked to background knowledge.
- Every collected chat was checked by an in-house evaluator for instruction adherence and coherence.
- 41.2% of Speaker 2 responses came from Reviews, 34.6% from Comments, 16.1% from Plots, and 8.1% from the Fact Table.
4 Models
The paper evaluates three model families that differ in how they use background resources: hierarchical generation without resources, copying with fallback generation, and direct span prediction.
- The resource is formed by merging comments, reviews, plots, and facts into one background-knowledge document.
- HRED encodes utterances hierarchically and generates responses from the resulting conversation-context representation without background knowledge.
- GTTP uses a hybrid pointer-generator that copies from the resource when appropriate and otherwise generates, given the document and recent context.
- BiDAF treats the conversation context as a question and predicts the answer span from the background resource.
- The selected copy-and-generate model includes a background encoder and a hybrid copy-or-generate decoder suited to this task.
5 Experimental Setup
The experiments define movie-conversation training instances, resource-construction methods, data splits, evaluation metrics, and human-judgment procedures. Span-based models face a 256-word resource constraint, while evaluation includes multiple references and both automatic and human measures.
- Data splits: Chats are split roughly 80%-10%-10% into train, validation, and test sets, with each movie assigned to exactly one split.This prevents movies in validation or test from appearing during training.
- Training instances: Each training instance contains the resource, the previous two utterances plus the current utterance as context, and an even-numbered response.A 10-turn chat therefore yields five training instances, and models predict the even-numbered responses.
- Resource construction: The background information is merged into one resource document, but BiDAF is restricted to 256 words because longer documents cause out-of-memory errors.The authors identify this as a severe limitation for BiDAF and other span-based models.
- Resource construction: The oracle, mixed-short, and mixed-long resource constructions retain the actual response span while differing in which source resources are included and how much text is retained.Mixed resources preserve each constituent source approximately in proportion to its length.
- Evaluation: Generation models are evaluated with BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L, while BiDAF additionally uses span-based F1.For BiDAF, the automatic metrics compare the predicted span with the reference span.
- Evaluation: Human evaluation uses 100 test responses per model and setup, rating fluency, relevance, humanness, and specificity; the dataset also provides three reference responses for each Speaker 2 utterance.Multiple references address the problem that several responses may be correct in the same context.
6 Results and Discussion
The experiments compare generation, copy-and-generate, and span-prediction models under different background-resource settings. Results favor exploiting relevant background text, while exposing scalability and coherence limitations in current span-based and copying approaches.
- Generation versus span prediction: BiDAF outperforms HRED and GTTP in the oracle setting, indicating that reproducing and lightly modifying background sentences can outperform generating responses from scratch.The authors describe current generation-based natural-language-generation capabilities as inadequate even in generate-or-copy modes.
- Limitations: BiDAF does not scale to longer documents, could not be run in the mixed-long setting, and requires models that can alter selected spans for contextual coherence.The authors also note that predicted spans are unnatural, with GTTP (o) receiving a higher human-likeness score than both BiDAF models.
- Effect of including background knowledge: Relevant background knowledge substantially improves GTTP over HRED, whereas mixed-long background knowledge yields little difference from the no-knowledge HRED baseline.HRED produces generic responses, with a specificity score of 2.06; the authors suggest selecting the relevant resource before span selection.
- Oracle versus mixed-short resource: BiDAF (ms) performs better than BiDAF (o) despite equal 256-word resource lengths, although the authors treat its proposed noise-filtering explanation as a hypothesis.GTTP (o) and GTTP (ms) have comparable poor performance, and GTTP lacks co-attention to filter noisy sentences.
- Copy-and-generate observations: 82.18% of tokens are copied in oracle copy-and-generate outputs, compared with 71.95% in mixed-long outputs, but copied spans are often contextually irrelevant.The model readily copies long contiguous sequences one word at a time, yet automatic metrics reveal relevance problems.
- Multiple-reference evaluation: Performance improves when multiple references are used, supporting evaluation methods that account for several dissimilar correct responses.Table 3 reports single-reference and multi-reference scores for the proposed models.
7 Conclusion
The paper introduces a background-linked dialog dataset and evaluates generation, copy-and-generate, and span-prediction paradigms. Results indicate that standard sequence-to-sequence NLG remains inadequate, while span-based models show promise but need improvement.
- The dataset explicitly links each response to background knowledge, unlike datasets containing only utterance sequences.
- Models are evaluated across generation-based, generate-or-copy, and span-prediction paradigms.
- Existing sequence-to-sequence NLG remains far from desirable, while span-based models show promise with clear scope for improvement.
- The authors propose future hybrid models combining span prediction with generation.
Model details - GTTP
GTTP represents both the background resource and dialog context, then combines vocabulary generation with resource copying. Its copy probability controls whether the next word is generated or copied from the resource.
- GTTP takes a resource-document and conversation context as input and produces a response sequence.The context includes previous utterances and the current utterance.
- The resource representation is computed as an attention-weighted sum of word representations.
- The context representation is likewise formed from attention-weighted representations of the previous utterances.
- At each decoding step, the model uses resource, context, and decoder-state representations to compute vocabulary probabilities and a generation probability.
- The final word distribution combines vocabulary generation with attention weights over document words.
Example from the multiple reference test set
The multiple-reference example shows that a single chat context can support several distinct but valid responses, differing in sentiment, movie details, and personal opinion.
- The “Secret Life of Pets” chat has multiple valid Speaker 2 responses for the same context.
- The references vary between negative or neutral critique assessments, favorite-character details, and personal opinions.
Hyper-parameters
The experiments use distinct optimizers, recurrent architectures, hidden-state sizes, dropout settings, embeddings, and vocabulary constraints for HRED, GTTP, and BiDAF. Examples also compare model outputs and dataset variants, while the appendix documents collection interfaces.
- Hyperparameters: HRED uses Adam, 300-dimensional GRUs, 0.25 dropout, 300-dimensional GloVe embeddings, and a 20,000-word vocabulary.
- Hyperparameters: GTTP uses Adagrad, 256-dimensional LSTMs, gradient clipping at maximum norm 2, and early stopping based on validation loss.
- Hyperparameters: BiDAF uses AdaDelta, 256-dimensional LSTMs, 0.2 dropout, 100-dimensional GloVe embeddings, and restricted context length.
- Model examples: HRED lacks background knowledge and fails to produce responses corresponding to the movie or context, whereas prediction-based models can produce appropriate recommendations through better context-document mapping.
- Model examples: The examples show that span-only outputs may be copied but unnatural, while complex backstory questions expose limited word-overlap-based understanding.
- Documentation: The appendix includes model details, hyperparameters, multiple-reference examples, sample outputs, and data-collection instruction screens.