Wizard of Wikipedia: Knowledge-Powered Conversational agents

1 INTRODUCTION

• Sequence-to-sequence models generally struggle to bring memory and knowledge to bear, limiting their ability to converse intelligently about a topic.The authors argue that direct knowledge-memory mechanisms are needed.Source: §1 · p. 1
• Transformer Memory Networks combine Memory Networks for knowledge retrieval and reading with Transformers for text representations and response generation.The architectures condition on retrieved knowledge and include both retrieval and generative versions.Source: §1 · p. 1
• A human wizard links the knowledge used in conversation to sentences from existing articles, providing a natural basis for training agents with memory that recalls and grounds on that text.This also provides a natural way to evaluate whether models can locate and use such knowledge.Source: §1 · p. 2
• The retrieval and generative Transformer Memory Network models produce engaging, knowledgeable conversations in automatic and human evaluations compared with baselines.The public benchmark is intended to encourage and measure further progress.Source: §1 · p. 2

2 RELATED WORK

• Open-Subtitles (Vinyals & Le, 2015), Persona-Chat, and Twitter (Sordoni et al., 2015) test dialogue-history attention without recalling long-term knowledge beyond model weights.These datasets do not explicitly study knowledge use.Source: §2 · p. 2
• Unlike goal-directed airline or restaurant dialogue, which typically accesses a database through API calls, this work studies unstructured knowledge across diverse topics potentially spanning all of Wikipedia.Examples of goal-directed dialogue include airline (El Asri et al., 2017) and restaurant booking (Wen et al., 2016; Bordes et al., 2017).Source: §2 · p. 2
• SQuAD and Open-SQuAD address factual answers with given-paragraph attention or Wikipedia retrieval, while QuAC frames question answering as dialogue; this work instead targets natural human dialogue.Its dialogue contains diverse utterances rather than only questions and answers.Source: §2 · p. 2
• Earlier knowledge-grounded open-domain dialogue used structured knowledge bases or unstructured articles and tips, but none authored dialogue with the supplied knowledge, leaving its usefulness unclear.Examples include movie discussions (Dodge et al., 2016), news (Parthasarathi & Pineau, 2018), and local-business dialogues (Ghazvininejad et al., 2018).Source: §2 · p. 2
• This work knows the Wikipedia articles and sentences grounding crowdworker dialogues, compares Memory Networks and Transformers, and combines these approaches for full multi-turn open-domain dialogue.Concurrently, Moghe et al. (2018) proposed a closed-domain movie-chat dataset; the paper reports models for full multi-turn open-domain dialogue, which it says had not previously been shown.Source: §2 · p. 2

Apprentice

• The task pairs a curious apprentice with a knowledgeable wizard in open-domain chit-chat whose topic can change naturally.The apprentice is encouraged to explore topics in depth, making the conversation more knowledge-focused than shallow chit-chat.Source: §3 · pp. 2–3

Wizard; Knowledge Selection and Response Generation

• Before replying, the wizard reviews retrieved Wikipedia passages and can select one relevant sentence or indicate that no sentence was used.Only one article and one sentence can be selected per turn; the wizard is instructed to use knowledge naturally rather than simply parrot it.Source: §3 · p. 3

Conversation Flow

• Either participant chooses the topic and speaks first; after each apprentice message, the wizard selects knowledge and replies until someone ends the chat.Chats last at least four or five turns per participant, with the minimum chosen randomly in advance.Source: §3 · p. 3

Topics

• The researchers crowd-sourced 1,365 natural open-domain topics, each linked to a Wikipedia article.The topics span varied subjects, including commuting, Gouda cheese, music festivals, podcasts, bowling, and Arnold Schwarzenegger.Source: §3 · p. 3

Knowledge Retrieval

• A fixed Open-SQuAD retriever supplies the wizard with relevant Wikipedia context using term-vector scoring over TF-IDF bag-of-word and n-gram vectors.It retrieves the top seven articles’ first paragraphs for the last two dialogue turns and the topic article’s first ten sentences; a better retriever could be learned for testing (Chen et al., 2017).Source: §3 · p. 3

Final Dialogue Dataset

• The corpus contains 22,311 dialogues and 201,999 turns, split into 166,787 train, 17,715 validation, and 17,497 test turns.Test Seen covers 533 topics overlapping with training, while Test Unseen covers 58 topics absent from train and validation.Source: §3 · p. 4, Table 1 · p. 4

4 MODELS

• The shared pipeline retrieves knowledge conditioned on dialogue history, attends over candidate sentences, and predicts the next utterance turn by turn.Models receive the topic and prior dialogue turns as context; responses may be selected from training utterances or generated word by word.Source: §4 · p. 4
• The retriever uses information retrieval to narrow a hierarchical Wikipedia memory to candidate sentences for fine-grained neural selection.It queries the topic and last two turns separately, retrieves the top seven articles’ first paragraphs, and prepends each sentence with its article title.Source: §4 · p. 4

4.1 RETRIEVAL TRANSFORMER MEMORY NETWORK

• The retrieval Transformer combines attended knowledge and dialogue encodings, then selects the candidate response with the highest similarity to that representation.Candidate responses are encoded separately with a Transformer.Source: §4.1 · p. 5

4.2 GENERATIVE TRANSFORMER MEMORY NETWORK

• Generative models select the most relevant knowledge, concatenate it with dialogue context for decoding, and use beam search of five with BPE encoding.The end-to-end version shares an encoder across knowledge candidates and dialogue, selects one candidate, and trains on response negative log-likelihood; wizard-selected knowledge can add a weighted cross-entropy loss.Source: §4.2 · p. 5
• The two-stage generator trains knowledge selection and utterance prediction separately, and knowledge dropout helps the decoder tolerate selection errors and speeds training.The method reduces generator access to knowledge for a fraction of training time.Source: §4.2 · pp. 5–6

5.1 KNOWLEDGE SELECTION TASK

• Transformers pretrained on Reddit work best at predicting wizard-selected knowledge, while multitasking on SQuAD has marginal impact; the best Transformer is used in the Two-stage model.The selection task conditions predictions on dialogue history and informs the choice of models for the two-stage architecture.Source: §5.1 · p. 6

5.2 FULL TASK: DIALOGUE WITH KNOWLEDGE

• Adding knowledge raises predicted-knowledge R@1 from 56 to 71 for Bow MemNet and from 79 to 87 for Transformer MemNet; gold knowledge improves performance dramatically.Models are evaluated by Recall@1 among 99 random response candidates and unigram F1; the pretrained Transformer Memory Network performs best in Table 3.Source: §5.2 · p. 6, Table 3 · p. 7
• With predicted knowledge, the Two-stage model outperforms End-to-end on perplexity and F1, while End-to-end leads with gold knowledge; auxiliary selection supervision improves every metric and knowledge dropout also helps.The results suggest that Two-stage benefits from its knowledge-selection module, while End-to-end is better at employing selected knowledge.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 HUMAN EVALUATION

• Retrieval models significantly outperform generative models in human engagingness (p < .05); adding knowledge does not significantly change retriever engagingness but raises Wiki F1 on seen and unseen sets.Human–model conversations covered a chosen topic for 3–5 turns; workers rated engagingness from 1–5, and Wiki F1 measured overlap with the topic’s Wikipedia page.Source: Table 5 · p. 7, §5.3 · p. 8
• For generative models, knowledge significantly improves engagingness (p < .01), and their Wiki F1 exceeds both knowledge-free counterparts and retrieval models on seen and unseen topics.The gap over retrieval models is larger on unseen topics, where retrievers cannot produce responses outside the training set.Source: Table 5 · p. 7, §5.3 · pp. 8–9
• All models still receive considerably lower human ratings than human-human conversations.The human evaluation comprised 546 conversations with ratings from 464 distinct workers.Source: §5.3 · pp. 8–9

• Transformer Memory Network architectures retrieve and attend to encyclopedic knowledge, then produce responses in retrieval or generative modes.Source: §6 · p. 9
• The Wizard of Wikipedia dataset grounds open-domain dialogues in Wikipedia knowledge; automatic and human experiments demonstrate model effectiveness, and the public benchmark invites further exploration.Source: §6 · p. 9
• Future work includes balancing retrieval responses’ engagingness with generative models’ ability to use new knowledge, jointly learning retrieval and reasoning, and connecting grounded dialogue with QA.These research strands aim to produce an engaging and knowledgeable conversational agent.Source: §6 · p. 9

A.2 WIZARD OF WIKIPEDIA EXAMPLES

• Wizards retrieved about 61 Wikipedia knowledge candidates per turn from dialogue history and clicked no sentence 6.2% of the time.Source: Figure 3 · p. 13
• Apprentices asked questions in 13.9% of training utterances, answered questions in 39.5%, and made new or follow-on statements in 49.3%.Source: Figure 3 · p. 13

A.3 TOPIC SELECTION

• Annotators mapped persona-interest sentences to relevant Wikipedia pages, producing 1,431 topics; sentences that were difficult to connect were left unlabeled.The persona source was Persona-Chat, whose roughly 1,000 personas each contained 4–5 sentences.Source: §A.3 · pp. 13–14
• The dataset retained persona topic sets and presented 2–3 related topic choices as conversation starters for each dialogue.Source: §A.3 · p. 14

B.1 KNOWLEDGE SELECTION

• The results indicate that the retrieval system could be improved, while the auxiliary loss clearly helps generative models.Table 6 reports test performance on knowledge-selection tasks.Source: Table 6 · p. 14
• The evaluation tests full-dialogue models on knowledge selection, with knowledge attention representing retrieval-system performance.Source: Table 6 · p. 14

C ERROR ANALYSIS

• The error analysis samples 20 conversations per experimental setting, split between seen and unseen topics, and analyzes lowercased, re-tokenized conversations single-blind.The analysis notes common errors and behaviors across the conversations.Source: C ERROR ANALYSIS · p. 14
• Human-human conversations feature more small talk and use topics as icebreakers, whereas dataset wizard conversations and all models tend toward fact-grounded wizard behavior.Some human participants treat bots as question-answer machines, suggesting potential benefit from SQuAD-like training data.Source: C ERROR ANALYSIS · p. 14
• Without knowledge, the retriever often produces non sequiturs and changes topics, especially on unseen conversations; with knowledge, it sticks to the chosen topic but struggles when people switch subjects.The knowledge-conditioned retriever may produce factually inaccurate answers, including listing Greek locations in response to a question about places to visit in Ireland, despite often using inviting responses.Source: C ERROR ANALYSIS · pp. 14–15

B.3 HUMAN EXPERIMENTS

• The human-evaluation Wiki F1 comparison uses a setup in which the wizard had direct access to Wikipedia passages in the UI, unlike the human-human evaluation setup.The table caption says this access explains the higher Wiki F1 values for both the wizard and apprentice, but provides no numerical values here.Source: Table 8 · p. 15

C ERROR ANALYSIS

• The generator without knowledge exhibits local and global repetition and inconsistent personality, while the knowledge-conditioned generator has fewer repetition problems by copying large Wikipedia fragments.The knowledge-conditioned generator may be selfish, provide incorrect dates, names, or words, and use formulaic responses.Source: C ERROR ANALYSIS · pp. 15–16
• The knowledge-conditioned generator generally produces accurate statements and successfully generalizes to unseen topics using Wikipedia knowledge.Its responses can still include incorrect details and formulaic phrasing.Source: C ERROR ANALYSIS · pp. 15–16