Source-linked AI summary
Dialog-based Language Learning
Jason Weston
TL;DR
This paper addresses the mismatch between language learning through communication and NLP training based on fixed labeled examples. It evaluates dialogue-based supervision tasks and learning strategies, including forward prediction of teacher replies; the approach shows promise, including when no reward signal is available.
Problem
Most NLP tasks in the paper’s framing use fixed word-, segment-, or sentence-level labels, whereas the authors study language learning from natural dialogue feedback.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The work presents evaluation datasets and models as a step toward learners that humans can teach through dialogue.
Source: §6 · p. 8The study covers only positive feedback and several forms of correction, leaving other dialogue replies and more complex settings with delayed rewards for future study.
Source: §6 · p. 9
Abstract
from arXiv · showhide
A long-term goal of machine learning research is to build an intelligent dialog agent. Most research in natural language understanding has focused on learning from fixed training sets of labeled data, with supervision either at the word level (tagging, parsing tasks) or sentence level (question answering, machine translation). This kind of supervision is not realistic of how humans learn, where language is both learned by, and used for, communication. In this work, we study dialog-based language learning, where supervision is given naturally and implicitly in the response of the dialog partner during the conversation. We study this setup in two domains: the bAbI dataset of (Weston et al., 2015) and large-scale question answering from (Dodge et al., 2015). We evaluate a set of baseline learning strategies on these tasks, and show that a novel model incorporating predictive lookahead is a promising approach for learning from a teacher's response. In particular, a surprising result is that it can learn to answer questions correctly without any reward-based supervision at all.
1 Introduction
The paper studies learning language through dialogue, where a partner’s replies provide natural feedback rather than fixed labeled examples. It introduces evaluation tasks and learning strategies, including forward prediction, which shows promise even without reward signals.
1 Introduction
- Human dialogue replies can provide informative feedback for learning language from the consequences of utterances.The introduction contrasts this with annotator-provided labels and notes that teacher replies may reward successful communication or correct unsuccessful attempts.Source: §1 · p. 1
- The paper targets the gap between dialogue-based learning and NLP tasks trained on word-, segment-, or sentence-level labels.It calls for evaluation datasets and learning algorithms suited to learning during conversation.Source: §1 · p. 1
- The work introduces natural-feedback tasks, evaluates baseline models against supervised learning, and proposes forward prediction of teacher replies, with promising results even without reward signals.Source: §1 · p. 1
2 Related Work
Related work spans feedback in human learning, supervised dialogue and question answering, and reinforcement learning. The authors distinguish their use of dialogue replies from reward-based feedback and note that forward prediction had not, to their knowledge, been used for dialogue.
2 Related Work
- Human-learning literature emphasizes social interaction, while the usefulness of feedback for grammar learning is disputed; support is reported in second-language learning and student learning.Source: §2 · p. 2
- Prior work used supervised dialogue and question answering, but those tasks typically did not consider feedback; other work used dialogue to collect knowledge, learn rules, or train parsers.Source: §2 · p. 2
- Reinforcement-learning approaches to dialogue often treat feedback as reward rather than exploiting the dialogue feedback itself.The cited work also notes reinforcement-learning applications in several other language and text tasks.Source: §2 · p. 2
- Forward prediction had been used in domains including eye tracking and robot control, but the authors were unaware of its prior use for dialogue.Source: §2 · p. 2
3 Dialog-Based Supervision Tasks
The section defines ten dialogue-supervision tasks across bAbI and MovieQA, ranging from observing an expert to receiving answers, hints, or other replies from a teacher. The tasks vary the information and reward available to learners, including setups with partial or no external reward.
3 Dialog-Based Supervision Tasks
- Ten supervision modes are built for bAbI and MovieQA, with experiments varying an imperfect answering policy’s accuracy across 0.5, 0.1, and 0.01.The datasets provide training, validation, and test sets; bAbI has 1,000, 100, and 1,000 questions, while MovieQA has approximately 96k, 10k, and 10k.Source: §3 · pp. 2, 4, 5
- Task 1 asks learners to imitate an expert student, while Task 2 provides positive or negative teacher feedback and an external reward for positive responses.Task 2 may provide little positive feedback when a learner answers incorrectly often.Source: §3 · pp. 2–3
- Task 3 supplies the correct answer after an error, whereas Task 4 gives a hint such as the correct answer’s class.The hints provide partial information and are described as intermediate between yes/no feedback and the full correction.Source: §3 · pp. 3–4
- Task 5 explains an incorrect answer by citing a known fact that supports the true answer and may contradict the learner’s response.Source: §3 · p. 4
- Task 6 gives external rewards for correct answers only 50% of the time, while Task 7 removes external rewards and retains text replies; learning to answer is possible in some conditions.Source: §3 · p. 4
- Task 8 combines observing expert dialogue with the learner’s own attempts to converse.Source: §3 · p. 4
- Task 9 has learners ask for the correct answer after an error, while Task 10 has them request a supporting-fact hint instead.The relevant information in Task 9 is spread over a larger dialogue context; Task 10’s hint resembles the supporting-fact feedback of Task 5.Source: §3 · p. 4
- The fixed answering policy is independent of the model being learned, so the experiments are not a reinforcement-learning setting.MovieQA includes an approximately 85k-fact knowledge base used by the memory network to retrieve question-relevant memories.Source: §3 · pp. 4–5
4 Learning Models
The section evaluates four dialog-learning strategies using an end-to-end memory network. It contrasts imitation objectives with forward prediction of the teacher’s response and a joint strategy.
- The shared memory network uses the last utterance and dialog context to read memory through attention hops and produce a candidate answer.Source: §4 · p. 5
- Imitation learning trains the network with cross-entropy to reproduce a speaker’s response given the last utterance and context.Source: §4 · p. 6
- Reward-based imitation applies that objective only to rewarded actions and discards the other actions from training.Source: §4 · p. 6
- Forward prediction uses the learner’s answer to predict the teacher’s next utterance, the changed state following that action.Source: §4 · p. 6
- The forward-prediction model retains the original input-and-context computation, then adds attention over candidate answers that incorporates which answer was selected.Source: §4 · p. 6
- After forward-prediction training, removing its added layers recovers the original memory network for answer prediction at test time.Source: §4 · p. 7
- The joint strategy shares weights between reward-based imitation and forward prediction, training with both criteria for each action.Source: §4 · p. 7
5 Experiments
Across the evaluated tasks, combining reward-based imitation (RBI) and forward prediction (FP) yields the best overall results. The experiments also show that FP can learn without rewards, while its performance depends on the feedback and task setting.
- Imitation is hard to beat when copying an expert, but performs poorly when imitating inaccurate answers, such as when πacc < 0.5.Source: §5 · p. 8
- RBI performs better when rewards are available, particularly on bAbI, but degrades when rewards are sparse, such as at πacc = 0.01.Source: §5 · p. 8
- FP is more robust across πacc levels, but RBI outperforms it on Tasks 1, 2, and 8, where FP lacks supervised learning or does not use rewards.Source: §5 · p. 8
- FP uses useful dialog feedback in Tasks 3–5, and full answer corrections help more than hints, which help more than yes/no feedback.Source: §5 · p. 8
- FP learns adequately on Task 7 without feedback, whereas RBI lacks positive examples; missing positive feedback in Task 6 particularly harms RBI.Source: §5 · p. 8
- FP has harder results on Tasks 9 and 10, where the question does not immediately precede the feedback.Source: §5 · p. 8
- Combining RBI and FP ameliorates the failings of each, yielding the best overall results.Source: §5 · p. 8
- FP obtains 68% accuracy on bAbI Task 2 under a biased policy, though it performs less well than under the earlier policy.The biased policy sometimes substitutes “bathroom” for incorrect answers.Source: §5 · p. 8
6 Conclusion
The paper presents evaluation datasets and models for dialog-based language learning as a step toward agents that humans can teach through conversation. Its evaluation is limited to positive feedback and several kinds of correction.