Source-linked AI summary
End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning
Jason D. Williams, Geoffrey Zweig
TL;DR
Task-oriented dialog systems require state representations, action policies, business rules, and access to external APIs. This paper combines an LSTM controller with developer software and trains it using supervised demonstrations followed by reinforcement learning. The experiments show that a supervised policy learned from few dialogs provides a reasonable starting point and substantially accelerates reinforcement-learning optimization.
Problem
Task-oriented dialog control traditionally requires substantial hand-crafting of dialog state, while systems must also represent business rules and interact with domain APIs.
Method
The method combines an LSTM that maps raw dialog history and extracted entities to action distributions with developer code for entity tracking, business rules, and API access, optimized by supervised and reinforcement learning.
Results
Supervised learning yields a reasonable policy from a small number of training dialogs, and this initial policy substantially accelerates reinforcement-learning optimization.
Takeaways & Limitations
Supervised demonstrations and reinforcement learning are complementary stages for end-to-end task-oriented dialog control.
Takeaways & Limitations
The approach delegates user-goal tracking to developer-provided code and assumes a simple name/value store when entity extraction is reliable.
Abstract
from arXiv · showhide
This paper presents a model for end-to-end learning of task-oriented dialog systems. The main component of the model is a recurrent neural network (an LSTM), which maps from raw dialog history directly to a distribution over system actions. The LSTM automatically infers a representation of dialog history, which relieves the system developer of much of the manual feature engineering of dialog state. In addition, the developer can provide software that expresses business rules and provides access to programmatic APIs, enabling the LSTM to take actions in the real world on behalf of the user. The LSTM can be optimized using supervised learning (SL), where a domain expert provides example dialogs which the LSTM should imitate; or using reinforcement learning (RL), where the system improves by interacting directly with end users. Experiments show that SL and RL are complementary: SL alone can derive a reasonable initial policy from a small number of training dialogs; and starting RL optimization with a policy trained with SL substantially accelerates the learning rate of RL.
1 Introduction
The paper proposes a staged framework for building task-oriented dialog bots, progressing from developer-defined actions and expert demonstrations to online improvement through user interaction. Its LSTM policy supports both supervised imitation and reinforcement learning, with real-time retraining for corrections.
- Framework: The framework mirrors human training: define available actions and rules, imitate expert dialogs, correct the bot, then improve from user interactions.Developers provide text actions, API calls, and action-masking code; domain experts provide examples and corrections before large-scale deployment.
- Model: The model combines a recurrent neural network with domain-specific software that encodes business rules and exposes APIs for actions such as ordering taxis or reserving tables.The network maps raw user turns and extracted entities directly to actions while inferring its own dialog-state representation.
- Optimization: Supervised learning imitates good dialogs, whereas reinforcement learning tries new action sequences and improves from a weak whole-dialog success signal.The two optimization modes use the same neural network but differ in the training signal.
- Optimization: The neural network can be retrained in under one second, enabling corrections during a conversation in real time.
2 Model description
The operational loop converts user input and extracted entities into developer-informed features, LSTM action probabilities, and executable text or API actions. Developer code constrains available actions and feeds resulting information back into the recurrent state.
- Input and features: The loop begins with user text, applies entity extraction and developer-provided entity resolution, and forms a feature vector from recognized entities and domain-specific features.The feature vector can encode recognized entity types and database-matching information.
- Neural policy: The LSTM consumes the feature vector, updates its internal state, and outputs probabilities over template actions with entity values replaced by placeholders.Template actions include responses such as “Do you want to call <name>?”.
- Action constraints: Developer code supplies an action mask that removes actions forbidden at the current timestep, after which the remaining probabilities are renormalized.For example, placing a phone call can be masked until a target number is identified.
- Action selection: The system samples an action during reinforcement learning but selects the highest-probability action when reinforcement learning is inactive.
- Execution loop: Selected actions are passed back to the LSTM and developer code, which either substitutes entities into text or invokes an API and returns its features for the next timestep.The operational loop therefore supports both user-facing text responses and external programmatic actions.
3 Related work
The paper contrasts its end-to-end dialog controller with prior systems that hand-craft state representations or learn action selection separately. Its central distinction is combining automatically inferred dialog history with explicit entity tracking and both supervised and reinforcement optimization.
- State tracking: Prior task-oriented systems commonly hand-crafted representations of user goals and dialog history, including in POMDP, supervised, hybrid, and commercial frameworks.
- State tracking: The proposed recurrent network automatically infers a dialog-history representation for predicting future actions, reducing the labor of designing an effective state space.User-goal tracking remains delegated to developer-provided code to support back-end database integration.
- State tracking: The method assumes a simple name/value store can track user goals when entity extraction is reliable, as in text interfaces without speech-recognition errors.
- Action selection: Existing action-selection approaches include hand-crafted rules, supervised learning from example dialogs, and reinforcement learning from whole-dialog rewards.These approaches trade off business-rule control, correction through additional examples, and learning directly from interaction.
- Combined optimization: The proposed network can be optimized with both supervised and reinforcement learning, while developer action masks encode business rules and preserve training-dialog behavior during reinforcement learning.
- Contribution: The paper presents this combination as an end-to-end dialog-control method that infers dialog history while explicitly tracking entities.
4 Example dialog task
The example task is a phone-call dialog system built around address-book contacts, varied names, and phone-number types. Its implementation combines entity definitions, programmatic logic, and example dialogs for supervised learning.
- The system initiates phone calls to address-book contacts from the Microsoft internal employee directory.
- Contacts may have name synonyms and multiple phone numbers, whose types also have synonyms such as “cell” for “mobile”.
- The system defines user entities for names, phone types, and yes/no responses, plus canonical and available-phone-type entities for system responses.
- The supervised-learning corpus contains 21 example dialogs covering variations in contact matches, available phone types, specified phone types, and unavailable requests.Dialogs average 7.0 turns, ranging from 4 to 11 turns, and use 14 action templates.
- A hand-designed stochastic simulated user can vary names and phone types, use nicknames, ignore or answer questions, add information, or give up.
5 Optimizing with supervised learning
Supervised learning evaluates how well an LSTM generalizes from small dialog sets, whether recurrence matters, and whether its confidence scores support active learning. Accuracy improves with more dialogs, recurrence resolves history-dependent distinctions, and low-confidence predictions help target corrections.
- Prediction accuracy: The 21-fold leave-one-out evaluation trains on 1, 2, 5, 10, or 20 dialogs and tests each model on the held-out dialog.
- Prediction accuracy: 70% of dialog turns are correctly predicted after one dialog, rising to over 90% after 20 dialogs, while nearly 50% of dialogs are completely correct.The authors state this is insufficient for final deployment but adequate for preliminary testing after a small number of dialogs.
- Benefit of recurrency: The DNN cannot reconstruct the full 20-dialog training set because identical local features can require different actions under different histories.The RNN also reconstructs the training set, motivating further study of recurrent architectures.
- Active learning: 80% of the 20 lowest-scored actions are incorrect even though incorrect actions comprise only 15% of turns.The result indicates that labeling low-scoring actions can rapidly correct errors.
- Active learning: The ROC analysis evaluates action scores by plotting false-positive rate against true-positive rate as the score threshold changes.
- Active learning: Retraining takes less than 1 second on a standard PC without a GPU, supporting frequent updates during active learning.
6 Optimizing with reinforcement learning
The paper evaluates reinforcement-learning optimization of an LSTM dialog policy, including how supervised pre-training affects learning. Results show that a few supervised dialogs accelerate learning and reduce performance variability.
- Reinforcement-learning method: RL explores action sequences using dialog-level rewards and updates the policy with a policy-gradient method.The return is compared with a baseline estimate of average recent performance when weighting gradient updates.
- Reinforcement-learning method: Natural-gradient optimization was rejected because inverting the neural network’s model-weight matrix was computationally intractable.
- Evaluation: The evaluation compares randomly initialized policies with policies pre-trained on 1, 2, 5, or 10 example dialogs before RL.Policies were evaluated after RL updates with a simulated user across 10 runs.
- Results: RL alone sometimes fails to discover a complete policy, producing lower average task completion and high variance.The difficulty is associated with discovering long action sequences under delayed rewards.
- Results: Adding a few supervised dialogs accelerates RL learning on average and reduces variability in the resulting policy.
7 Conclusion
The paper presents an end-to-end task-oriented dialog system in which an LSTM maps raw dialog history to action distributions while developer software handles entities, APIs, and business rules. Experiments show that supervised learning from few dialogs provides a reasonable policy and substantially accelerates later reinforcement-learning optimization.
- The LSTM infers dialog-state representations from raw history, reducing the need for hand-crafted state representations.
- Developer code tracks entities, wraps external API calls, and enforces business rules on the policy.
- Supervised learning yields a reasonable policy from a small number of training dialogs and substantially accelerates reinforcement-learning optimization.
- The paper presents a first demonstration of end-to-end learning for task-oriented dialog control.
A Example dialogs
The example dialogs illustrate task-oriented phone-calling interactions involving contact lookup, phone-type selection, ambiguity resolution, unavailable numbers, and confirmation of actions.
- The dialogs include straightforward calls in which the system identifies a contact and selects a requested phone type.
- When a requested phone type is unavailable, the system offers an available alternative and proceeds only after the user confirms.
- The system handles refusal of an alternative by ending the interaction without placing the call.
- The examples cover ambiguous names and unknown names by requesting a full name or asking the user to try again.