PaperlayerView exact revision on arXiv

Source-linked AI summary

Dialogue Learning With Human-In-The-Loop

Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, Jason Weston

arXiv:1611.09823v3cs.AIcs.CL

TL;DR

Conversational agents are commonly trained on fixed labeled datasets, leaving online improvement from dialogue-partner feedback less studied. This paper trains question-answering bots with reinforcement learning from teacher feedback, and finds that online and batch learning can work, including with real human feedback.

  • Problem

    Most research focused on fixed labeled datasets rather than how a conversational agent can improve through online interaction with a teacher.

    Source: §1 · p. 1

  • Method

    The paper casts question answering as reinforcement learning, training a MemN2N policy from teacher rewards and textual feedback in simulated and human dialogue.

    Source: §4.2 · p. 4

  • Results

    Both fully online and iterative batch policy learning are viable, and forward prediction works online and with real human feedback.

    Source: §6 · p. 10

  • Takeaways & Limitations

    A fixed-data-trained model can feasibly continue learning from human interactions in a (semi-)online pipeline.

    Source: §1 · p. 1, §6 · p. 10

    The comparison with Weston uses a different initialization regime: its synthetic labels are balanced, while this work updates a policy from randomly initialized weights.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.

1 INTRODUCTION

The paper addresses the gap between fixed-dataset training and conversational agents that improve through online teacher feedback. It studies reinforcement-learning approaches to question answering and reports the feasibility of a pipeline that learns from human interactions after fixed-data training.

  • The bot answers teacher questions about a short story or facts and receives numerical rewards, textual feedback, or both.Source: §1 · p. 1
  • The paper studies both simulated dialogue and real-human feedback collected through Amazon Mechanical Turk.Source: §1 · p. 1
  • The authors find it feasible to start with a fixed-data-trained model and improve it through interactions with humans.Source: §1 · p. 1
  • The paper uses “student” for a human or bot learner and “teacher” for its more knowledgeable dialogue partner.Source: §1 · p. 1

2 RELATED WORK

Prior dialogue reinforcement-learning and question-answering work leaves a gap around end-to-end agents that learn policies from online interaction and teacher feedback. The paper distinguishes its setting from fixed-policy or fixed-dataset approaches and adds experiments with real language feedback.

  • Earlier dialogue reinforcement learning mainly addressed slot-filling systems, whereas this work studies end-to-end learning and textual feedback beyond reward-based setups.Source: §2 · p. 2
  • Dialogue question-answering work used fixed datasets with gold annotations and did not train through reinforcement learning on the teacher’s response.Source: §2 · p. 2
  • Weston (2016) used fixed policies and simulated language; this work trains policies across iterations and tests feedback from real humans.Source: §2 · p. 2

3 DATASET AND TASKS

The paper adapts question-answering datasets for simulated teacher-bot interaction and adds a Mechanical Turk setting with natural human questions and feedback. The simulator updates the bot’s policy during learning, so its actions affect the data collected in later episodes or batches.

3.1 SIMULATOR

  • The simulator adapts bAbI stories and WikiMovies questions into dialogues where a scripted teacher asks, the bot answers, and the teacher gives feedback.Source: §3.1 · p. 3
  • Task 6 gives positive textual feedback for correct answers, but positive reward only half the time; wrong answers receive textual feedback containing the answer.Source: §3.1 · p. 3, Figure 1 · p. 3
  • Unlike fixed tasks, the simulator updates the policy after feedback and deploys it in the next episode or batch, making the policy affect subsequent training data.Source: §3.1 · p. 3

3.2 MECHANICAL TURK EXPERIMENTS

  • Mechanical Turk replaces templated questions and feedback with natural human utterances, whose lexical variability is greater and less easily categorized.Source: §3.2 · p. 4

4 METHODS

The methods use MemN2N as the dialogue policy and train it online from teacher feedback through reward-based imitation, REINFORCE, or textual-feedback prediction.

4.1 MODEL ARCHITECTURE

  • MemN2N reads dialogue history and context memories through repeated attention hops, then predicts among candidate answers.Its memories can encode dialogue history and knowledge-base facts.Source: §4.1 · p. 4

4.2 REINFORCEMENT LEARNING

  • The contextual-bandit setup treats dialogue history as state, candidate answers as actions, and teacher correctness as binary reward; updates can follow online or dataset-sized batches.The authors report robustness across batch sizes, alleviating the need for correction terms.Source: §4.2 · pp. 4–5

4.2.1 REWARD-BASED IMITATION (RBI)

  • RBI imitates positively rewarded answers using cross-entropy and ignores other answers; online exploration uses an ϵ-greedy strategy.The bot chooses its highest-probability answer with probability 1 − ϵ and a random answer with probability ϵ.Source: §4.2.1 · p. 5

4.2.2 REINFORCE

  • REINFORCE maximizes sampled teacher reward using an independent learned baseline; unlike RBI, it also uses incorrect behavior and explores from its action distribution.The baseline predicts future reward from the memory network output after its last hop, and its error is not backpropagated through the policy.Source: §4.2.2 · p. 5

4.2.3 FORWARD PREDICTION (FP)

  • When numerical rewards are unavailable, FP predicts textual teacher feedback conditioned on dialogue context and the selected answer.For online learning, the paper adds ϵ-greedy answer exploration and balances training across clusters of teacher responses. Balancing is intended to prevent dominant response classes from degenerating learning into input-independent output.Source: §4.2.3 · p. 6

5 EXPERIMENTS

Simulator experiments show that online feedback can improve question answering, though exploration or balancing is needed in some settings for stable learning. Mechanical Turk experiments find that textual feedback and rewards work best together.

5.1 SIMULATOR

  • Random exploration can improve RBI and FP; performance is largely independent of batch size, and RBI performs similarly to REINFORCE.Source: §5.1 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 8
  • Across dataset iterations, RBI improves without random exploration; the authors attribute this to variety in positive-reward examples from the initial randomly initialized policy.Source: §5.1 · p. 8
  • FP becomes unstable when negative responses grow scarce, but balancing or sufficient random exploration helps, and RBI+FP alleviates the instability without either.Without enough negative responses, FP can collapse into repeatedly predicting the same responses before recovering and collapsing again.Source: §5.1 · p. 8
  • 98% accuracy at iteration 6 exceeded fixed-policy RBI at πacc 1% (59%) and 10% (81%), but not at 50% (99%).The comparison is for bAbI Task 6; the fixed-policy results use pre-built datasets.Source: Table 1 · p. 9, §5.1 · p. 9

5.2 HUMAN FEEDBACK

  • With Mechanical Turk feedback, combining FP and RBI outperforms either alone; FP remains useful with sparse rewards, while RBI then uses only the initial labeled examples.The table covers feedback on 10,000 model predictions, with sparse binary rewards added to textual feedback.Source: Table 2 · p. 10, §5.2 · p. 10
  • In additional experiments, results with human feedback were competitive with synthetic feedback and the fully supervised case; the protocol collected textual feedback on bot's responses for an additional 10,000 questions.Source: §5.2 · pp. 9–10

6 CONCLUSION

The paper finds that textual-feedback and reward-based dialogue learning can work in fully online or iterative-batch settings, with instability taken into account. It reports that forward prediction works online and with real human feedback, and that a semi-online pipeline can improve a model initialized on fixed data.

  • Fully online and iterative-batch policy learning are viable when potential learning instabilities are taken into account.Source: §6 · p. 10
  • Forward prediction works in both online learning and learning from real human feedback.Source: §6 · p. 10
  • A practical pipeline can start from a fixed-data-trained model and improve it through human interactions in a semi-online fashion.Source: §6 · p. 10
  • The paper identifies never-ending learning as a direction for future research.Source: §6 · p. 10

A FURTHER SIMULATOR TASK DETAILS

The appendix describes ten simulator tasks for different teacher-feedback setups, and reports results on Tasks 2, 3, and 4. Unlike fixed datasets, the simulator lets the learner update after feedback and apply the update in the next episode.

A FURTHER SIMULATOR TASK DETAILS

  • Tasks 1–5 cover direct imitation, positive or negative textual feedback, answers supplied after errors, answer-class hints, and explanations identifying supporting facts, respectively.These are tasks introduced in Weston (2016).Source: A FURTHER SIMULATOR TASK DETAILS · p. 13
  • Tasks 6–10 vary feedback through intermittent positive rewards, text without rewards, combined supervision, student questions about errors, and hints after requests for help.These are tasks introduced in Weston (2016).Source: A FURTHER SIMULATOR TASK DETAILS · p. 13
  • The simulator trains learners on-the-fly, so feedback or rewards can update the model for the next episode; appendix results cover Tasks 2, 3, and 4.Source: A FURTHER SIMULATOR TASK DETAILS · p. 13

B INSTRUCTIONS GIVEN TO TURKERS

The Turk instructions ask workers to respond to a student's answer using the supplied question and correct answers. They also specify response-quality guidance, while Figure 5 illustrates simulator tasks with varied teacher feedback and binary feedback.

B INSTRUCTIONS GIVEN TO TURKERS

  • Workers received a teacher question, its correct answer or answers, and a proposed student reply as each task's three-part input.Source: B INSTRUCTIONS GIVEN TO TURKERS · p. 13
  • Workers were asked to give brief feedback using the supplied correct answers to judge the reply; examples contrast approving “red” with correcting “green.”Source: B INSTRUCTIONS GIVEN TO TURKERS · p. 13
  • Instructions encouraged varied responses and few spelling mistakes, prohibited addressing the student or class directly, and mentioned possible bonuses for quality.Source: B INSTRUCTIONS GIVEN TO TURKERS · p. 13
  • Figure 5 illustrates ten tasks evaluating teacher-response forms and binary feedback, with examples of correct and incorrect student answers.Source: Figure 5 · p. 14

C ADDITIONAL EXPERIMENTS

The additional experiments vary exploration and batch size across bAbI and WikiMovies tasks, with figure captions reporting robustness to batch size for several WikiMovies settings. The supplied table caption identifies a dataset-sized-batch bAbI Task 3 accuracy comparison but provides no model scores.

C ADDITIONAL EXPERIMENTS

C.1 ADDITIONAL EXPERIMENTS FOR MECHANICAL TURK SETUP

Mechanical Turk experiments compare real human feedback with synthetic feedback and fully supervised imitation learning.

C.1 ADDITIONAL EXPERIMENTS FOR MECHANICAL TURK SETUP

  • RBI+FP with synthetic Task 2+3 or Task 3 feedback outperforms real feedback, while real feedback outperforms synthetic Task 2 feedback.Real feedback remains competitive despite its lexical and semantic variability.Source: §C.1 · p. 22, Table 4 · p. 22
  • RBI+FP and FP alone approach the performance of fully supervised imitation learning on human questions.The supervised comparison uses turker-authored questions with gold labels and no numerical rewards or textual feedback.Source: §C.1 · p. 22, Table 5 · p. 22

C.2 SECOND ITERATION OF FEEDBACK

A second feedback iteration adds 10,000 examples collected using the first-iteration model as predictor. Performance improves overall, with only slight variation across reward fractions and exploration settings.

C.2 SECOND ITERATION OF FEEDBACK

  • The second iteration adds 10,000 examples collected using the first-iteration RBI+FP model with r = 1, whose test accuracy was 0.478.Training continues on the original 1k+10k examples plus the additional 10k.Source: §C.2 · pp. 22–23
  • Performance improves in the second iteration, with slightly better results for large r and ϵ = 0.5.The improvement is mostly invariant to these parameters, likely because FP uses feedback from incorrect predictions.Source: §C.2 · p. 23
Loading 1611.09823v3…