Source-linked AI summary
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, Larry Heck
TL;DR
Offline supervised training and interactive reinforcement learning can expose task-oriented dialogue agents to different dialogue-state distributions. The paper combines end-to-end neural modeling with user-guided imitation and reinforcement learning, finding that agents learn from mistakes through teaching and improve task completion after subsequent reinforcement learning.
Problem
Supervised pre-training followed by interactive reinforcement learning may suffer from dialogue-state distribution mismatch, making unfamiliar states difficult for the agent to recover from.
Method
The paper trains an end-to-end neural task-oriented dialogue agent with supervised pre-training, user-demonstrated corrective actions, imitation learning, and subsequent reinforcement learning.
Results
The agent learns effectively from mistakes through imitation learning from user teaching, while subsequent reinforcement learning further improves successful task completion.
Takeaways & Limitations
User teaching can adapt the neural dialogue agent to interactive dialogue-state distributions, after which reinforcement learning further improves its task performance.
Abstract
from arXiv · showhide
In this work, we present a hybrid learning method for training task-oriented dialogue systems through online user interactions. Popular methods for learning task-oriented dialogues include applying reinforcement learning with user feedback on supervised pre-training models. Efficiency of such learning method may suffer from the mismatch of dialogue state distribution between offline training and online interactive learning stages. To address this challenge, we propose a hybrid imitation and reinforcement learning method, with which a dialogue agent can effectively learn from its interaction with users by learning from human teaching and feedback. We design a neural network based task-oriented dialogue agent that can be optimized end-to-end with the proposed learning method. Experimental results show that our end-to-end dialogue agent can learn effectively from the mistake it makes via imitation learning from user teaching. Applying reinforcement learning with user feedback after the imitation learning stage further improves the agent's capability in successfully completing a task.
1 Introduction
Conventional task-oriented dialogue systems rely on modular pipelines whose independently optimized components can misalign and amplify errors. The paper proposes end-to-end hybrid imitation and reinforcement learning, using user corrections to address dialogue-state distribution mismatch during interaction.
- 1 Introduction: Modular task-oriented dialogue pipelines independently train NLU, dialogue-state tracking, and policy components, whose objectives may not align with task success or user satisfaction.Errors from upstream modules can propagate to downstream components and become amplified.
- 1 Introduction: End-to-end neural systems have been studied with both supervised learning and deep reinforcement learning for task-oriented dialogue.Unlike single-turn chit-chat training, task-oriented dialogue requires multi-turn reasoning and planning.
- 1 Introduction: Offline supervised pre-training can mismatch the dialogue-state distributions encountered during interactive reinforcement learning.An agent’s response changes future states, so a small mistake may lead to an unfamiliar state from which recovery is difficult.
- 1 Introduction: The proposed hybrid method has users demonstrate corrective actions after agent mistakes, adds those guided dialogues to the corpus, and fine-tunes the policy before reinforcement learning.The method combines imitation from user teaching with reinforcement learning from interaction.
- 1 Introduction: The paper designs a neural task-oriented dialogue system optimized end-to-end across natural language understanding, dialogue-state tracking, and policy learning.The stated contributions include both the end-to-end architecture and the hybrid training method.
2 Related Work
Prior work explored POMDP-based policy learning, end-to-end neural models, and supervised-plus-reinforcement learning for task-oriented dialogue. This paper distinguishes its approach by targeting the dialogue-state distribution mismatch between supervised pre-training and interactive learning.
- 2 Related Work: POMDP-based reinforcement learning requires carefully designed dialogue states and actions, limiting policy learning to restricted domains.The design requirement is presented as necessary for tractable policy learning.
- 2 Related Work: End-to-end supervised models were evaluated on fixed dialogue corpora, leaving their generalization to unseen interactive dialogue states unknown.The paper positions its own supervised-plus-reinforcement-learning approach as addressing this interactive-learning concern.
- 2 Related Work: Prior end-to-end systems used memory, query-regression, gated-memory, and copy-augmented networks to learn dialogue state or select responses.These approaches include models that condition response selection on dialogue context.
- 2 Related Work: Soft knowledge-base lookup in an end-to-end reinforcement-learning agent is described as vulnerable to entity updates and additions in real-world information systems.The paper instead uses symbolic queries and leaves knowledge-base entity selection to external services.
- 2 Related Work: A prior hybrid supervised-and-reinforcement-learning system improved learning speed but did not address supervised-to-interactive dialogue-state distribution mismatch.The present framework identifies that mismatch as its target problem.
3 Proposed Method
The proposed system connects neural components for utterance encoding, dialogue state tracking, policy learning, knowledge-base interaction, and response generation into one differentiable end-to-end model. Training combines supervised learning, user teaching through corrected mistakes, and reinforcement learning from user feedback.
- End-to-end architecture: The hierarchical LSTM architecture encodes user utterances and previous system actions into a continuous dialogue state used for tracking and policy decisions.A bidirectional utterance-level LSTM produces the user representation, while a dialogue-level LSTM maintains state across turns.
- Dialogue state and KB operation: The dialogue state tracker estimates slot-value distributions, which are converted into symbolic knowledge-base queries for retrieving task-relevant information.Queries can use the best hypothesis or an n-best list of candidate slot values, while entity ranking is delegated to an external recommender.
- Dialogue policy: The policy network selects dialogue acts from the dialogue state, belief-tracker log probabilities, and encoded query-result summaries.A single-hidden-layer MLP with softmax activation produces the system action, which is then realized through template-based natural-language generation.
- End-to-end training: Connecting the components yields a fully differentiable system whose utterance encoder, dialogue state tracker, and policy network can be trained end-to-end.The supervised objective combines cross-entropy losses for dialogue-state tracking and system-action prediction.
- Human teaching and feedback: Interactive training addresses offline-to-online state-distribution mismatch by adding expert corrections from agent-generated dialogues to the training corpus.The updated model is retrained on the aggregated corpus before further optimization with user feedback and reinforcement learning.
- Reinforcement learning: Reinforcement learning further optimizes the policy using user feedback, a step penalty for shorter dialogues, and REINFORCE with a softmax sampling policy.The reward design uses task completion and can be extended with interaction naturalness or knowledge-base query costs.
4 Experiments
The experiments evaluate supervised and interactive learning on restaurant-search and movie-booking dialogues, measuring tracking, task success, dialogue length, and human-rated quality. Imitation learning from user teaching improves adaptation and task success, while subsequent end-to-end reinforcement learning provides further gains.
- Experimental setup: Experiments use DSTC2 restaurant-search data and a movie-booking corpus, with extended natural-language forms used to test generalization.The movie-booking corpus contains 100K training dialogues and 10K each for development and testing; its simulator supports varied user personalities.
- Supervised learning results: The model achieves near-state-of-the-art dialogue-state tracking on DSTC2 and promising individual-slot and joint-slot accuracy on movie booking.DSTC2 evaluation uses live ASR transcriptions, whereas movie-booking evaluation uses text input, which partly explains the higher accuracy in that domain.
- Interactive evaluation: Interactive evaluation measures task success rate, average dialogue-turn size, and dialogue-state tracking accuracy, while also comparing end-to-end and policy-only reinforcement learning.Figures 3–5 show learning curves for the three metrics; Figure 6 compares the two reinforcement-learning settings on task success.
- Task success and dialogue length: Imitation learning improves task success after supervised pre-training, and reinforcement learning further increases success; reinforcement learning alone remains at a very low level after 10K simulations.The supervised model performs poorly under new user templates, while imitation-plus-reinforcement learning handles more challenging tasks with higher success but slightly longer dialogues than supervised-plus-reinforcement learning.
- Dialogue state tracking: Dialogue-state tracking accuracy rises from 50.51% under interactive evaluation to 67.47% after 500 imitation-learning sessions, while subsequent reinforcement learning further improves tracking.The fixed-corpus supervised joint-slot accuracy is 84.57%, but introducing new NLG templates reduces it to 50.51% during interactive evaluation.
- End-to-end optimization and human evaluation: End-to-end reinforcement-learning updates achieve higher dialogue task success than updating only the policy network, and interactive imitation-plus-reinforcement learning improves human-rated model quality.Human judges rated 100 dialogues for each of three models, with three judges scoring every system turn.
5 Conclusions
The proposed hybrid approach trains an end-to-end task-oriented dialogue agent through supervised learning, user teaching, imitation learning, and reinforcement learning. The agent learns from mistakes, while subsequent user-feedback reinforcement learning further improves performance, including dialogue state tracking.
- The hybrid method combines supervised pre-training with imitation learning from user teaching and reinforcement learning from user feedback.
- The end-to-end neural dialogue agent learns from mistakes made during interaction with users.
- Applying reinforcement learning after imitation learning further improves model performance on dialogue policy and dialogue state tracking.