Learning through Dialogue Interactions by Asking Questions

1 INTRODUCTION

• The work targets a gap between human learners, who can ask questions when confused, and dialogue agents trained mainly on fixed answers.Confusion may involve an unfamiliar surface form, a semantically complicated sentence, or an unknown word.Source: §1 · p. 1
• Interaction can help a learner clarify a question, identify relevant known facts, or acquire missing knowledge for future use.A student may write down knowledge supplied by the teacher when it was absent from the student’s knowledge sources.Source: §1 · p. 1
• The learning system improves through interaction in both offline supervised and online reinforcement learning settings.The approach is also validated with human teachers on Amazon Mechanical Turk, where similar results are observed.Source: §1 · pp. 1–2
• The authors note that their mistake categories are non-exhaustive and do not address failures in dialogue generation.Source: §1 · p. 1

2 RELATED WORK

• Earlier interactive language-learning work includes language games and studies of feedback in language learning and education.Source: §2 · p. 2
• Neural dialogue research includes chit-chat, goal-oriented systems, and supervised question answering, while most systems learn from fixed signals rather than user interaction.Source: §2 · p. 2
• Weston (2016) introduced dialogue learning schemes, but did not fully explore learning through questioning and interaction.The authors describe their work as a natural extension of that research.Source: §2 · p. 2

3 THE TASKS

• The task suite covers question clarification, knowledge operation over an existing knowledge base, and acquisition of missing knowledge.Experiments adapt WikiMovies, with roughly 100,000 questions over 75,000 entities; accuracy measures the percentage of correct answers.Source: §3 · p. 2

3.1 QUESTION CLARIFICATION.

• For clarification, the bot encounters previously unseen misspellings and can ask for a paraphrase or verify its interpretation before answering.The authors choose spelling mistakes for ease of dataset construction, although other language misunderstandings are possible.Source: §3.1 · p. 3, §3.3 · p. 3

3.2 KNOWLEDGE OPERATION

• For knowledge operation, the bot has relevant facts but can ask the teacher to identify a relevant fact or verify a candidate fact to support reasoning.Source: §3.2 · p. 3

3.3 KNOWLEDGE ACQUISITION

• For knowledge acquisition, the bot’s incomplete KB omits information needed to answer, so it asks the teacher and later must answer a repeated question using what it learned.The model must retain the missing entity and connect it to known entities; the answer having appeared earlier in dialogue does not make the task trivial.Source: §3.3 · p. 3
• Acquisition tasks vary which information is hidden: the question entity, answer entity, relation, their connecting triple, or all of them.Source: §3.3 · pp. 4–5

4.1 SIMULATOR

• The simulator supports two aims: testing whether asking questions helps across tasks and whether students can learn when to ask, using offline supervised learning and online RL.Source: §4.1 · p. 6

4.1.1 OFFLINE SUPERVISED LEARNING

• Offline training uses dialogues from a fixed student policy that answers correctly half the time; its TrainQA, TrainAQ, and TrainMix sets respectively omit, always include, or mix question asking.Testing crosses those training sets with TestQA, TestAQ, and TestModelAQ; the latter additionally evaluates question choice in Question and Knowledge Verification tasks.Source: §4.1.1 · p. 6

4.1.2 ONLINE REINFORCEMENT LEARNING (RL)

• In online RL, the student learns whether to ask under a cost for each question, then answers; episode reward combines the asking cost with +1 for a correct answer or −1 otherwise.The student can ask at most one question, and the question type is fixed by the task.Source: §4.1.2 · p. 7
• RL evaluation varies student preparedness: Good-Students have complete relevant knowledge, Poor-Students have flawed questions or missing facts, and Medium-Students mix the two settings equally.Source: §4.1.2 · p. 7

4.2 MECHANICAL TURK DATA

• Mechanical Turk experiments collect real-language dialogues for Tasks 4 and 8, with about 10,000 training, 1,000 validation, and 2,500 test episodes per task.Human teachers replace the simulator and provide more varied language, while the bot’s dialogue actions remain essentially unchanged.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

• MemN2N takes the teacher’s question and short- and long-term memories as input, then outputs an answer label.Source: §5 · pp. 8–9
• Vanilla-MemN2N uses reward-based imitation: it maximizes likelihood of correct student answers and discards examples with incorrect final answers.Its candidate answers are words appearing in memory.Source: §5 · p. 9
• Cont-MemN2N represents each word using the average of its embedding and nearby-word embeddings, with context size selected on the development set.Source: §5 · p. 9
• TrainQA (+FP) jointly predicts correct bot answers and teacher feedback during training, addressing the other offline models’ omission of feedback; at test time it predicts only the bot’s answer.Source: §5 · p. 9
• Online RL uses separate policies for asking and answering, updated with REINFORCE from episode reward relative to a learned baseline.The baseline is an independent MemN2N trained to estimate cumulative reward.Source: §5 · p. 9
• Training first optimizes the final-answer policy alone, then trains the answer and question policies together.The authors report that this strategy yields better results; they select the best-reward model on the development set during the later training phase.Source: §5 · p. 9

6.1 SIMULATOR

• TrainQA+TestAQ underperforms TrainQA+TestQA on Question Clarification and Knowledge Operation, but improves most Knowledge Acquisition tasks, with one stated exception.TrainAQ+TestQA performs worst; the reported Knowledge Acquisition exception is Cont-MemN2N on Missing Triple.Source: §6.1 · p. 10
• TrainMix is slightly below TrainAQ+TestAQ but remains competitive on both test settings for Question Clarification and Knowledge Operations.The paper describes TrainMix as bridging datasets generated using QA and AQ.Source: §6.1 · p. 10
• Test-time questions help, and model-chosen questions perform similarly to correct questions; without test-time questions, accuracy is below 0.01 on three missing-knowledge tasks.Those tasks are Missing Question Entity, Missing Answer Entity, and Missing Everything.Source: §6.1 · p. 10
• Cont-MemN2N significantly outperforms vanilla-MemN2N, with context offering evidence to distinguish correct answers from dialogue-history candidates, especially for unfamiliar words.Source: §6.1 · p. 10
• In online learning, good students stop asking earlier as costs rise; poor and medium students ask longer, and their accuracy drops more when asking declines.The reported cost patterns distinguish Question Verification from Missing Answer Entities.Source: Figure 6 · p. 11, §6.1 · p. 11

6.2 MECHANICAL TURK

• Mechanical Turk performance is lower than on related simulator tasks, with lexical variety, complexity, noise, and a smaller training set offered as explanations.The real-data training set was 10k versus 180k for the simulator; the paper reports that real data was measurably superior to simulated data.Source: §6.2 · p. 11
• TrainAQ+TestAQ performs best across simulator settings and Mechanical Turk tasks, where asking humans lets the bot outperform one that only answers.The Mechanical Turk results follow the same trend despite lower performance than on the related simulator tasks.Source: §6.1 · p. 10, Table 3 · p. 12, §6.2 · p. 12

• Across online and offline settings, teaching a bot to interact with humans facilitates language understanding and, in most cases, improves question answering.Source: §7 · p. 12

Appendix

• MemN2N reads dialogue and knowledge-base memories through repeated weighted hops before predicting among candidate answers; experiments use three hops.Its memories encode both short-term dialogue history and long-term knowledge-base facts.Source: Appendix · p. 13
• The simulator adapts WikiMovies and gives students relevant KB facts, teacher follow-up interactions, and five unrelated dialogue pairs to test retention.The dataset contains roughly 100k questions over more than 75k entities.Source: Appendix · p. 14
• QA students never ask clarification questions, AQ students always ask scenario-dependent questions, and Mixed training combines both while requiring a final answer.Source: Appendix · p. 14
• Mechanical Turk tasks collect teacher responses to student questions, including relevance feedback and answers supplied when the student requests missing information.Source: Appendix · p. 15
• Combined real-and-simulated training performs best; real-only results exceed simulation-only on Tasks 4 (0.774 vs. 0.714) and 8 (0.797 vs. 0.788), while AQ beats QA across settings.The AQ advantage holds despite far fewer real training examples than simulated examples.Source: Appendix · pp. 15–16, Table 5 · p. 16, Table 4 · p. 16