Source-linked AI summary
Wizard of Wikipedia: Knowledge-Powered Conversational agents
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Weston
TL;DR
Open-domain dialogue systems often generate generic responses without explicitly using recalled knowledge, and the field lacks a large grounded benchmark. The paper builds Wikipedia-grounded conversations and Transformer Memory Network models that retrieve, attend to, and generate from knowledge. The models demonstrate knowledgeable dialogue in automatic and human evaluations, while retrieval models are more engaging than generative models and knowledgeable retrievers achieve higher Wiki F1 scores.
Problem
Open-domain dialogue lacks convincing demonstrations and a supervised benchmark for explicitly grounding responses in recalled knowledge.
Method
The paper collects Wikipedia-grounded human conversations and develops retrieval and generative Transformer Memory Networks that retrieve, read, and condition responses on knowledge.
Results
Retrieval models significantly outperform generative models on human engagingness, while knowledgeable retrievers obtain significantly higher Wiki F1 scores on seen and unseen test sets.
Takeaways & Limitations
The publicly released benchmark supports measuring further improvements in knowledgeable open-domain conversational agents.
Takeaways & Limitations
A considerable gap remains between human ratings of human-human conversations and ratings of all evaluated models.
Abstract
from arXiv · showhide
In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.
1 INTRODUCTION
The paper targets open-domain dialogue agents that use recalled knowledge rather than generating generic responses from memorized parameters. It introduces a Wikipedia-grounded conversation dataset and Transformer Memory Network architectures for retrieving, conditioning on, and generating knowledgeable responses.
- Open-domain dialogue requires agents to retrieve and use knowledge while maintaining engaging conversation.
- Transformer Memory Network retrieval and generative architectures are evaluated with automatic metrics and human judgments against standard Memory Networks and Transformers.
- 1365 discussion topics and 201,999 utterances form a supervised dataset of human conversations linked to Wikipedia knowledge.
- Wizard-linked Wikipedia sentences provide both grounding for training knowledgeable agents and evaluation of knowledge location and use.
- The benchmark is publicly released to measure further progress in knowledgeable open-domain dialogue.
2 RELATED WORK
Prior dialogue benchmarks generally emphasize dialogue history or structured databases rather than explicit access to broad, unstructured knowledge. This work is positioned closest to knowledge-grounded non-goal-directed dialogue using external text.
- Popular chit-chat datasets test sequence-to-sequence models over recent dialogue history without explicit long-term knowledge retrieval.
- Goal-directed dialogue commonly conditions on databases through APIs, whereas this work studies unstructured knowledge spanning diverse Wikipedia topics.
- Question answering uses retrieval and knowledge conditioning for factual answers, but does not generate responses from conversation history.
- Closest prior dialogue studies incorporate structured or unstructured external knowledge for domains such as movies, news, and local businesses.
3 WIZARD OF WIKIPEDIA
Wizard of Wikipedia collects open-domain conversations in which a knowledgeable wizard uses retrieved Wikipedia passages to inform responses to a curious apprentice. The resulting dataset provides grounded dialogues across diverse topics with seen and unseen test splits.
- The task pairs a knowledgeable wizard with a curious apprentice whose conversation topic can naturally change.
- The apprentice is instructed to explore topics deeply while keeping the conversation engaging, emphasizing knowledge use beyond shallow chit-chat.
- Before each turn, the wizard receives potentially relevant Wikipedia paragraphs and may base the response on that knowledge.
- The dialogue flow has the wizard select a relevant sentence, respond using it, and repeat until the chat ends after at least four or five turns each.
- 1365 diverse topics are linked to Wikipedia articles, including commuting, Gouda cheese, music festivals, podcasts, bowling, and Arnold Schwarzenegger.
- The fixed retriever uses TF-IDF bag-of-word and n-gram vectors to present top-7 article candidates from recent turns and the original topic.
- The dataset contains 22,311 dialogues and 201,999 turns, with Test Seen and Test Unseen subsets for overlapping and novel topics.
4 MODELS
The models retrieve a small set of Wikipedia candidates, attend to relevant knowledge, and predict the next utterance using either candidate selection or word-by-word generation. Generative variants differ in whether knowledge selection and response generation are trained separately or jointly.
- Transformer Memory Network models retrieve relevant information, attend to it, and generate responses grounded in dialogue history.
- Retrieval models select responses from training utterances, whereas generative models produce responses word by word.
- At each turn, the model receives the dialogue context and predicts the next utterance.
- Information retrieval reduces a large hierarchical memory to candidate passages before fine-grained neural knowledge selection.
- The retrieval architecture combines attended knowledge with dialogue encoding and scores candidate responses using Transformer representations.
- Generative models select knowledge and concatenate it with dialogue context before decoding, using either Two-stage or End-to-end training.
- Knowledge dropout prevents attention to knowledge during some training steps, improving generator resilience to knowledge-selection errors and speeding training.
5 EXPERIMENTS
The experiments evaluate knowledge selection and full knowledge-grounded dialogue, using retrieval and generative models under gold- and predicted-knowledge settings. Results show consistent benefits from knowledge, with different strengths across model architectures and human-evaluation dimensions.
- Experimental setup: The experiments first test knowledge selection from dialogue history, then evaluate full dialogue generation with either gold or predicted knowledge.The full-task evaluation separately considers retrieval and generative models.
- Retrieval experiments: Using knowledge improves retrieval performance, including Bow MemNet from 56 to 71 R@1 and Transformer MemNet from 79 to 87 R@1 with predicted knowledge.The retrieval task ranks the gold response among 99 candidates and also measures unigram F1.
- Generative experiments: The Two-stage generative model performs best with predicted knowledge, whereas the End-to-end model performs best with gold knowledge.The results suggest that Two-stage benefits from stronger knowledge selection, while End-to-end better employs selected knowledge.
- Generative experiments: Auxiliary knowledge-selection supervision improves the End-to-end model on every metric, and knowledge dropout further improves performance.Both generative models outperform their no-knowledge Transformer counterpart and improve substantially with gold knowledge.
- Human evaluation: Retrieval models significantly outperform generative models on human engagingness, while knowledgeable retrievers achieve significantly higher Wiki F1 on seen and unseen test sets.Engagingness differences between retrievers with and without knowledge are not significant.
- Human evaluation: Knowledge significantly improves generative-model engagingness ratings, and generative models convey more knowledge than retrieval models on seen and unseen sets.The gap between retrieval and generative models is larger on unseen data because retrieval models are limited to responses from the training set.
- Human evaluation: All evaluated models remain considerably below human-human ratings on engagingness.Figure 2 provides selected conversations between humans and models.
6 CONCLUSION
The paper presents knowledge-grounded open-domain dialogue agents built around Transformer Memory Networks and the Wizard of Wikipedia dataset. It concludes that the benchmark supports further exploration, while identifying several unresolved directions for achieving conversations that are both engaging and knowledgeable.
- Contributions: Transformer Memory Network agents retrieve and attend to encyclopedic knowledge before producing retrieval-based or generative responses.The models use large memory systems containing knowledge about the world.
- Contributions: The Wizard of Wikipedia dataset provides open-domain dialogues grounded in Wikipedia knowledge for training and evaluation.The paper reports effectiveness through automatic and human experiments.
- Implications: The publicly available benchmark is intended to encourage further model exploration in knowledge-grounded dialogue.The conclusion anticipates significant advances from continued work in this research direction.
- Future work: Future work includes bridging retrieval engagingness with generative generalization to new knowledge and topics, jointly learning retrieval and reasoning, and relating the task to question answering.The stated aim is an engaging and knowledgeable conversational agent.
A.2 WIZARD OF WIKIPEDIA EXAMPLES
The Wizard of Wikipedia dataset pairs human conversations with Wikipedia-grounded dialogue and varied conversational acts. Its examples show how personal discussion can connect to retrieved factual knowledge.
- Wizard–apprentice conversations combine personal dialogue with Wikipedia knowledge about the discussion topic.A lifeguard exchange is paired with retrieved information about lifeguards’ rescue, swimming, and first-aid roles.
- Each utterance retrieves approximately 61 knowledge candidates from the dialogue history, while wizards select no sentence in 6.2% of cases.
- The dataset includes questions, answers, and new or follow-on statements, reflecting varied dialogue acts.Apprentices ask questions in 13.9% of training utterances, answer questions in 39.5%, and make other statements in 49.3%.
B ADDITIONAL EXPERIMENTS
Additional experiments assess knowledge selection within the full dialogue task and indicate that auxiliary supervision benefits generative models.
- The retrieval system can be improved, while an auxiliary loss clearly helps the generative models.The table evaluates knowledge-selection performance and the full-dialogue models’ knowledge attention.
C ERROR ANALYSIS
Error analysis contrasts human and model conversations and identifies distinct retrieval and generation errors. Knowledge improves topicality, factuality, repetition, and generalization, but introduces conversational and factual weaknesses.
- Human–model differences: Human–human conversations contain more small talk, whereas Wizard conversations become more grounded in factual sentences.Models also tend to adopt the wizard role and produce more factual sentences than humans in the comparison conversations.
- Human–model differences: Humans sometimes treat the bot as a question-answer machine, suggesting that SQuAD-like training data could improve the models.
- Retriever errors: Without knowledge, the retriever often produces non sequiturs and changes topics rapidly, especially on unseen conversations.
- Retriever errors: With knowledge, the retriever stays on topic more strongly but struggles when the human changes subjects and can generate factually inaccurate answers.One example answers a question about visiting Ireland by listing locations in Greece.
- Generator errors: Without knowledge, the generator shows repetition and personality inconsistencies, while knowledge reduces repetition but can produce copied, selfish, formulaic, or date-inaccurate responses.
- Generator errors: The knowledge-equipped generator can generalize successfully to unseen topics using information from Wikipedia.