Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Braden Hancock; Antoine Bordes; Pierre-Emmanuel Mazaré; Jason Weston

Source-linked AI summary

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, Jason Weston

arXiv:1901.05415v4Jan 16, 2019cs.CLcs.AIcs.HCcs.LGstat.ML

TL;DR

Dialogue agents often leave abundant post-deployment conversations unused because conventional supervision is costly and may differ from deployment. This paper proposes a self-feeding chatbot that estimates satisfaction, imitates satisfied responses, and predicts feedback after suspected mistakes. On PersonaChat, automatically extracted examples improve dialogue ability regardless of available supervised data, with the strongest improvement from combining dialogue and feedback examples.

Problem
Dialogue agents typically rely on expensive supervised conversations, while abundant post-deployment interactions remain an underused source of task-specific training signal.
Method
The self-feeding chatbot predicts user satisfaction, turns satisfied user responses into DIALOGUE examples, and turns natural-language corrective responses into FEEDBACK examples for auxiliary training.
Results
Automatically extracted DIALOGUE and FEEDBACK examples improve PersonaChat dialogue ability regardless of the amount of supervised data, with the largest improvement when both are added.
Takeaways & Limitations
The approach supports continued dialogue improvement from natural user responses without special feedback structure, numerical rewards, or additional human intervention.
Takeaways & Limitations
The FEEDBACK examples are used without inspection, post-processing, or cleaning, relying on a learnable relationship between contexts and natural user feedback.

Abstract

from arXiv · show

The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in conversation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user's responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot's dialogue abilities further. On the PersonaChat chit-chat dataset with over 131k training examples, we find that learning from dialogue with a self-feeding chatbot significantly improves performance, regardless of the amount of traditional supervision.

1 Introduction

The paper proposes a self-feeding chatbot that learns from conversations after deployment while addressing the risks of learning from its own mistakes. It extracts dialogue and feedback examples using predicted user satisfaction, and reports improved dialogue ability on PersonaChat regardless of available supervised data.

Motivation: Extensive, costly supervision is commonly used to train dialogue agents, but deployment conversations are abundant, task-specific, dynamic, and cheap.Existing supervised conversations may also differ substantially from deployment environments.
Motivation: Naively training on a chatbot’s own outputs can reinforce failure modes and produce conversations unlike the target domain.Requesting feedback can also fail when the model is poor at recognizing its own mistakes or requires structured rewards.
Approach: The self-feeding chatbot extracts new training examples from deployment conversations by estimating partner satisfaction and choosing between imitation and feedback collection.Satisfied user responses become DIALOGUE targets, while responses to a feedback request become FEEDBACK examples.
Results: On PersonaChat, adding automatically extracted DIALOGUE examples, FEEDBACK examples, or both improves the chatbot regardless of the number of supervised examples, with both types performing best.The paper also reports that satisfaction classification significantly outperforms an uncertainty-based approach and releases three deployment-related datasets.
Approach: Dialogue ability improves when the chatbot imitates responses received during satisfaction and predicts natural-language feedback after suspected mistakes.The new examples require no special structure, numerical reward, or additional human intervention to use.

2 Related Work

The related work connects self-feeding dialogue to lifelong learning, active learning, feedback-based question answering, and mistake detection. The paper distinguishes its focus on learning when to ask for feedback and using natural dialogue responses to improve the model.

Connections: The work relates to lifelong learning, never-ending language learning, active learning, and predictive modeling.These areas provide broader conceptual context for learning during deployment.
Feedback-based learning: Prior dialogue and question-answering studies explored active learning with verbal cues, scalar rewards, and forward prediction of teacher responses.The paper extends forward prediction by adding the ability to recognize when feedback should be requested and to use that feedback for learning.
Question asking: Related work on question asking primarily identifies which question to ask, whereas this paper focuses first on learning when to ask a question.Earlier work considered this timing question mainly in question-answering settings rather than open-ended dialogue.
Mistake detection: Mistake detection from user responses had been studied in virtual assistants, but those systems did not use identified mistakes to improve themselves.Other work found that judging chatbot-response appropriateness depends strongly on user responses rather than preceding context alone.
Feedback representations: Most other feedback-based dialogue approaches use scalar rewards or collect knowledge, rather than learning directly from dialogue messages.The paper contrasts these settings with its use of natural user responses as training signal.

3 The Self-Feeding Chatbot

The self-feeding chatbot learns from deployment conversations by routing turns into DIALOGUE or FEEDBACK examples according to predicted user satisfaction. Its training uses human-human data initially, then periodically retrains on naturally occurring human responses and feedback, with experiments bounded to comparable supervised and deployment-example counts.

Initial training: The initial agent is trained on DIALOGUE next-utterance prediction and SATISFACTION prediction using available Human-Human examples.Human-Human examples come from conversations between two humans.
Deployment data collection: When predicted satisfaction exceeds threshold t, the chatbot creates a Human-Bot DIALOGUE example from the previous context and the human’s response.The bot’s own utterance is not used as the target.
Deployment data collection: When predicted satisfaction falls below t, the chatbot asks what it should have said instead and uses the user’s response as a FEEDBACK example.The feedback task predicts the feedback associated with the preceding context, and the bot then resets its history to continue deployment.
Retraining and scope: The chatbot is periodically retrained on all available data, while the experiments limit supervised and deployment examples to the same order of magnitude.The authors envision deployment data eventually reaching 100× or more the amount of supervised data, but do not test that regime here.
Feedback data: Natural feedback is retained without inspection or cleaning because its varied forms have a learnable relationship with conversation contexts.Feedback may be a verbatim response, a description, or a list of possible responses.
Task definitions: DIALOGUE examples use context-response pairs for coherent, engaging conversation, while SATISFACTION predicts a speaking partner’s satisfaction from the conversation context.Satisfaction labels range from dissatisfied to satisfied, and the human response can reveal dissatisfaction more clearly than the bot’s utterance.

4 Model and Settings

The model combines task-specific neural components within a shared interface and trains DIALOGUE, SATISFACTION, and FEEDBACK tasks using separate batches and tuned loss scaling. The implementation uses Transformer-based models with task-specific evaluation and deployment candidate handling, while noting open task-structure questions and an informal validation practice.

Architecture: The chatbot has a shared interface for processing, history storage, candidate preparation, and control flow, plus a neural-network model for each task.Each model includes embeddings, a network body, and a task head.
Multi-task training: Each training batch contains examples from one task, candidate sets remain separate, and task-specific cross-entropy losses are scaled using validation-set factors.The factors account for differences in dataset size, loss magnitude, and dataset relevance.
Open question: The authors identify optimal task structure in multi-task learning architectures as an open research problem.
Model architecture: Transformer models encode SATISFACTION contexts into scalar predictions, while DIALOGUE and FEEDBACK use their respective task heads.The Transformer architecture is used as the underlying model family for the dialogue agent.
Candidate settings: Evaluation assigns each example 19 randomly sampled candidates, whereas deployment ranks the 127,712 unique HH DIALOGUE train-split candidates.The deployment candidates are encoded once and ranked for each context.
Input settings: The experiments use a maximum dialogue history of 2 and 300-dimensional fastText embeddings, with vocabulary sizes varying from 11.5k to 23.5k words.The history includes the agent’s previous utterance and its partner’s response.
Evaluation: Development used part of the DIALOGUE validation split as an informal test set, while the hidden test set was reserved for final reported numbers.This distinguishes development monitoring from the official final evaluation.

5 Experimental Results

Deployment examples improve dialogue performance across supervision levels, with complementary gains from HB DIALOGUE and FEEDBACK data. Satisfaction classification also outperforms uncertainty-based feedback requests.

Deployment examples improve DIALOGUE accuracy regardless of the number of supervised HH examples.
Up to 9.4 accuracy points, or 31%, are gained when the supervised HH dataset is small.
Even with 131k supervised examples, deployment data adds 1.6 accuracy points, while combining HB DIALOGUE and FEEDBACK provides complementary signal.
20k FEEDBACK examples benefit the agent about as much as 60k HB DIALOGUE examples.FEEDBACK examples target model mistakes, whereas HB DIALOGUE examples come from conversations where users appear satisfied.
The best model reaches 46.3 accuracy on DIALOGUE and 68.4 on FEEDBACK, suggesting FEEDBACK is the simpler task overall.
Fresher FEEDBACK data yields a statistically significant 0.4-point improvement and reduces the gap to fully supervised performance by 17%.The improvement modifies only 33% of the training data.
With 1k training examples, the satisfaction classifier exceeds uncertainty methods and a regular-expression baseline by up to 0.28 and 0.42 F1 points.

6 Future Work

The paper proposes extending self-feeding beyond satisfied-user imitation and unsatisfied-user feedback toward dialogue strategies that improve both performance and future data collection.

Future work could learn which questions elicit the most valuable feedback and intermix requests for FEEDBACK and SATISFACTION examples.The authors frame this as a metalearning direction for improving dialogue ability and the ability to improve further.

A Data Collection Protocol

The data-collection protocol iteratively deploys a chatbot, gathers satisfaction and feedback examples, retrains on selected data, and extracts HB DIALOGUE examples from conversation logs.

The initial 20k HH DIALOGUE model collected 2.5k SATISFACTION examples, split into 1k train, 500 validation, and 1k test examples.
After retraining with the 1k SATISFACTION training examples, the agent was deployed again to collect deployment examples.
The protocol collected 40k FEEDBACK examples across 17,250 conversations, then retrained using 20k of them before collecting another 20k.
The resulting dataset contained 60k FEEDBACK examples, while 60k HB DIALOGUE examples were extracted from deployment conversation logs.
An additional 40k SATISFACTION training examples supported the SATISFACTION learning-curve experiments.
No crowdworker-conversation filtering was performed, preserving a workflow that requires no developer intervention.

B Results with 10k Candidates

Deployment examples remain beneficial when the candidate-response set expands to 10,000, a setting closer to interactive dialogue.

Adding HB DIALOGUE and FEEDBACK examples continues to improve DIALOGUE performance at all supervision levels with 10,000 candidates.

C PERSONACHAT Comparisons and Baselines

The experiments use the original PERSONACHAT test set to compare model and baseline accuracy, without access to profile information.

The reported comparisons use the original PERSONACHAT test set rather than the newer ConvAI2 test set.
All reported models lack access to the profiles used to create the conversations.Models with profile access tend to perform better.
Table 6 reports the accuracy of various models and baselines on the original PERSONACHAT test set.

D Using Chatbot Responses as Targets

Using chatbot-generated responses as training targets decreased quality, while identifying reliably good chatbot responses proved difficult because appropriate responses resemble the dialogue task itself.

Adding chatbot responses as targets decreased quality with both 20k and 131k Human-Human DIALOGUE examples.The comparison included all bot responses and only responses with estimated satisfaction above 0.5.
0.70 was the maximum precision achievable at at least 10% recall when identifying highly rated chatbot responses from over 34k examples.Ratings of 5 were treated as positive, ratings of 1–3 as negative, and ratings of 4 were discarded.
Really good chatbot responses are difficult to identify because they usually resemble normal human-to-human conversation.Recognizing an appropriate next utterance is the DIALOGUE task the model is intended to solve.
Negative responses are more semantically similar because they commonly ask for clarification or convey confusion.

E The Effect of Data Freshness

Feedback examples collected after retraining were more valuable than earlier feedback examples, whereas the same freshness effect was not observed for Human-Bot dialogue examples.

Feedback set B collected from a more recently retrained model improved performance more than feedback set A.
The fresher feedback produced better performance when retraining occurred before collecting the next 20k examples (p = 0.027).Figure 3 shows 20k supervised DIALOGUE examples followed by two 20k feedback collections; shaded regions are 95% confidence intervals.
The experiments used regular expressions to identify user dissatisfaction and trigger feedback requests before training a satisfaction classifier.
Table 9 lists the hyperparameters used to obtain the results in Table 3.

H Data Collection Interfaces

The study collected deployment conversations and satisfaction ratings through crowdsourcing interfaces designed to simulate natural chatbot interactions, including explicit feedback after apparent mistakes.

Deployment interface: Crowdworkers in the deployment environment were instructed to talk as they would to a person they had just met.
Satisfaction interface: The SATISFACTION interface required workers to rate each chatbot response from 1 to 5 after their own response.Its environment was otherwise very similar to the deployment environment.
Deployment conversations: When the chatbot made an apparent mistake, it asked users what it could have said instead and used their replies as feedback.The sample deployment conversations show users correcting irrelevant, incoherent, or off-topic responses.
Deployment conversations: Some deployment samples show the chatbot continuing ordinary conversation after users accepted its responses.
Satisfaction conversations: Satisfaction examples included both highly rated conversational responses and responses users marked as nonsensical or irrelevant.Examples show rewards of 5 for acceptable responses and rewards of 1 for incorrect responses.