Source-linked AI summary
Continuously Learning Neural Dialogue Management
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, Steve Young
TL;DR
Task-oriented dialogue systems face limitations in both early-stage reinforcement learning and supervised imitation. This paper trains one neural network with supervised dialogue data and then refines it through reinforcement learning; experiments report improved outcomes in simulation and with real users, including mismatched conditions.
Problem
Reinforcement learning can perform poorly early in training, while supervised learning may miss future action effects, lack state coverage, and imitate nonoptimal wizard behavior.
Source: §1 · p. 1Method
Results
In simulation, success rate improved 1–8% across SER conditions during 6,000 training dialogues, and reinforcement learning improved quality with real users after 200 training dialogues.
Source: §3.2 · p. 4, §3.3 · p. 4Takeaways & Limitations
The model modified its behavior via delayed reward in a mismatched environment and achieved a better success rate, with only a few hundred supervised dialogue examples.
Source: §4 · p. 4Gradient descent is slow and has poor convergence properties.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
We describe a two-step approach for dialogue management in task-oriented spoken dialogue systems. A unified neural network framework is proposed to enable the system to first learn by supervision from a set of dialogue data and then continuously improve its behaviour via reinforcement learning, all using gradient-based algorithms on one single model. The experiments demonstrate the supervised model's effectiveness in the corpus-based evaluation, with user simulation, and with paid human subjects. The use of reinforcement learning further improves the model's performance in both interactive settings, especially under higher-noise conditions.
1 Introduction
Task-oriented dialogue management is difficult to handcraft, while reinforcement learning can perform poorly early in training and supervised learning has important behavioral and coverage limits. The paper proposes combining supervised pretraining with reinforcement learning in one network to address these complementary challenges.
1 Introduction
- Reinforcement learning frames dialogue management as trial-and-error optimization of a reward that may be delayed until dialogue success.This learning setup can use human interaction to optimize the dialogue policy.Source: §1 · p. 1
- Even sample-efficient reinforcement learning can perform poorly early in training, harming user experience and helping explain commercial systems’ reliance on handcrafted dialogue management.Sample-efficient algorithms can learn policies from a minimal number of dialogues, but early performance remains a concern.Source: §1 · p. 1
- Supervised dialogue learning ignores future effects of actions, may lack coverage of the large state space, and can imitate suboptimal human behavior, especially in noisy or larger domains.These limitations become more consequential when larger domains require multi-step planning.Source: §1 · p. 1
- A single network is first trained on supervised dialogue data and then improved through reinforcement learning, without changing the system architecture.The model uses a full action set rather than the constrained summary actions used by many RL dialogue systems.Source: §1 · p. 2
2 Neural Dialogue Management
The policy network maps a belief state and dialogue history to a master dialogue action. It is first trained to mimic corpus responses, then improved with reinforcement learning while retaining its supervised initialization.
2 Neural Dialogue Management
- The policy network uses a belief state as input and produces a master dialogue action, which is passed to the natural language generator.Its outputs include dialogue-act and query predictions alongside binary offer-slot predictions.Source: §2 · p. 2
2.1 Phase I: Supervised Learning
- Supervised training minimizes joint cross-entropy between action labels and model predictions to mimic responses in corpus data.Training data may come from Wizard-of-Oz collection or interactions with an existing policy.Source: §2.1 · p. 2
2.2 Phase II: Reinforcement Learning
- Reinforcement learning maximizes expected dialogue reward while keeping the policy network structure and initial weights fixed by supervised pretraining.The reward is accumulated across dialogue turns using a discount factor.Source: §2.2 · p. 3
- Natural Actor-Critic uses an episodic least-squares method to obtain a natural-gradient direction that approximately limits changes in the policy distribution.For larger models, a truncated variant can practically calculate the natural gradient.Source: §2.2 · p. 3
3 Experimental Results
In the Cambridge restaurant domain, the supervised policy performed well on corpus labels and could be further improved through reinforcement learning in simulated and human interactions. Simulation gains were greatest under higher semantic error rates, while human evaluation also favored the RL-trained policy.
3 Experimental Results
- The experiments used a live telephone-based restaurant-information system for Cambridge, with a domain of approximately 150 venues.Each venue had six slots: three search constraints and three informable properties available after finding a database entity.Source: §3 · p. 3
3.1 Supervised Learning on Corpus Data
- On the test set, the policy accurately predicted dialogue acts and generally provided the right information, while Query had lower weighted F-1.The authors suggest this may reflect training data with robust ASR results and more offers than queries.Source: §3.1 · p. 3, Table 1 · p. 3
3.2 Policy Network in Simulation
- Across semantic error rates, RL increased simulated success by 1–8% over 6,000 training dialogues, with the greatest improvement at higher error rates.The authors report that the model learned to double-check user intent when semantic hypotheses were more corrupted.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 Policy Network with Real Users
- Human evaluation indicated that the SL policy worked well with users and was improved by RL training on 200 dialogues.Evaluation used a further 110 dialogues with both SL-only and SL+RL policies.Source: §3.3 · p. 4
4 Conclusion
The paper presents a unified neural framework for dialogue management, trained first with supervised learning and then improved through reinforcement learning in simulated or spoken interactions. Experiments show better success rates in mismatched environments using a delayed reward signal.
4 Conclusion
- A unified neural framework trains dialogue management first on a fixed dataset with SL and then improves it through RL in simulated or spoken interactions.The experiments demonstrated efficiency with only a few hundred supervised dialogue examples.Source: §4 · p. 4
- In a mismatched environment, the model modified its behaviour via a delayed reward signal and achieved a better success rate.Source: §4 · p. 4
- The model was tested in both simulated and real-user settings after supervised training.Source: §4 · p. 4