Source-linked AI summary
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, Steve Young
TL;DR
Reliable reward estimation for dialogue-policy reinforcement learning is difficult because real-user feedback is noisy and costly, while user goals and suitable offline data are often unavailable. The paper jointly learns a policy and reward model online using active learning with Gaussian processes over unsupervised RNN dialogue embeddings. The approach supports stable policy optimisation, reduces feedback requests, and outperforms other state-of-the-art methods in the Cambridge restaurant domain.
Problem
Reliable reward estimation is difficult because real-user feedback is noisy and costly, while user goals and suitable offline data are often unavailable.
Method
The framework jointly trains a dialogue policy and reward estimator online using active learning, Gaussian process classification, a noise model, and unsupervised RNN-based dialogue embeddings.
Results
The model achieved efficient policy learning and better performance than other state-of-the-art methods in the Cambridge restaurant domain.
Takeaways & Limitations
The techniques enable online learning in deployed real-world dialogue systems without a large manually annotated corpus or a user simulator.
Takeaways & Limitations
The reward function focuses primarily on task success, which may be too simplistic for many commercial applications.
Abstract
from arXiv · showhide
The ability to compute an accurate reward function is essential for optimising a dialogue policy via reinforcement learning. In real-world applications, using explicit user feedback as the reward signal is often unreliable and costly to collect. This problem can be mitigated if the user's intent is known in advance or data is available to pre-train a task success predictor off-line. In practice neither of these apply for most real world applications. Here we propose an on-line learning framework whereby the dialogue policy is jointly trained alongside the reward model via active learning with a Gaussian process model. This Gaussian process operates on a continuous space dialogue representation generated in an unsupervised fashion using a recurrent neural network encoder-decoder. The experimental results demonstrate that the proposed framework is able to significantly reduce data annotation costs and mitigate noisy user feedback in dialogue policy learning.
1 Introduction
Task-oriented dialogue policies can be optimised with reinforcement learning, but reliable reward estimation is difficult when user goals are unknown and feedback is noisy or costly. The paper proposes jointly learning the policy and reward model online through active feedback, noise modelling, and learned dialogue representations.
- Reinforcement learning optimises dialogue management through trial and error using a potentially delayed reward objective.
- Reward functions commonly penalise each turn and assign a large positive reward when a pre-specified task succeeds.
- Objective reward estimation is impractical when the user’s goal is unavailable, while subjective feedback can be inaccurate, costly, and destabilising.
- Offline task-success predictors can avoid objective checks during online learning, but user simulators provide only rough real-user approximations and are costly to develop.
- The proposed framework jointly learns a dialogue policy and reward estimator online using active feedback requests, a noise model, Gaussian process classification, and RNN-based dialogue embeddings.
2 Related Work
Related work has explored dialogue evaluation, learned reward models, active annotation, inverse reinforcement learning, and preference-based reinforcement learning. These approaches commonly depend on annotations, demonstrations, or user judgements that can be difficult or costly to obtain in real applications.
- PARADISE inferred user satisfaction from task completion and dialogue features, and the measure was later used as a policy-learning reward.
- Reward-model approaches use collaborative filtering, reward shaping, or expert satisfaction ratings, but assume reliable dialogue annotations that are hard to obtain.
- Uncertainty models and active learning can guide annotation by identifying when feedback is needed and which samples are informative.
- Inverse reinforcement learning recovers a reward from demonstrations of good behaviour before learning a policy that maximises the recovered reward.
- Preference-based reinforcement learning uses relative dialogue judgements, but pairwise rankings remain costly and do not scale well in real applications.
3 Proposed Framework
The framework jointly learns a dialogue policy and a noise-robust reward model online, using fixed-dimensional dialogue embeddings and uncertainty-guided user feedback. An unsupervised recurrent encoder-decoder creates the representations consumed by Gaussian-process reward learning.
- System framework: The framework combines a dialogue policy, dialogue embedding function, and active reward model trained from user feedback.Each dialogue is mapped to a fixed-dimensional representation that serves as the reward model’s input.
- Online joint learning: The key contribution is jointly learning the noise-robust reward model and dialogue policy online, with the user acting as supervisor.Active learning reduces the supervision burden, while the embedding function is trained offline on corpus data.
- Dialogue embeddings: An unsupervised embedding function maps variable-length dialogues into a fixed-dimensional continuous space for reward modelling.The embedding function is trained with an encoder-decoder objective that reconstructs dialogue features.
- Dialogue embeddings: A BLSTM encoder processes turn-level features in both directions, while a forward LSTM decoder reconstructs the feature sequence from the dialogue representation.The dialogue representation is calculated from the encoder’s hidden sequences.
- Active reward learning: The Gaussian-process reward model estimates dialogue success and prediction uncertainty from the continuous dialogue representation.Success is modelled as binary classification with a probit-linked latent function and a noise kernel for inaccurate user ratings.
- Active reward learning: Active learning requests user feedback only when the Gaussian process is sufficiently uncertain about its success prediction.The decision boundary incorporates both posterior mean and variance; otherwise, the model’s predictive rating updates the policy directly.
4 Experimental results
In a Cambridge restaurant domain, the proposed on-line GP reward model used unsupervised dialogue embeddings and active feedback querying to support policy optimisation. It achieved strong subjective success while reducing feedback requests and mitigating noisy user ratings.
- Experimental setup: The live restaurant-information system used GP-SARSA policies, an LSTM encoder-decoder embedding, and a Gaussian-process reward model for real-user dialogues.Dialogue representations were built from turn-level features, while the GP estimated task success and uncertainty for reward modelling.
- 4.1 Dialogue representations: The 650 test-dialogue embeddings showed dialogue length and success-related structure, with successful dialogues averaging fewer than 10 turns.Long failed dialogues exceeding 15 turns clustered together, while other failures were more dispersed.
- 4.2 Dialogue Policy Learning: All four systems exceeded 80% subjective success after approximately 500 training dialogues, while the Obj=Subj system needed about 700 dialogues to obtain 500 usable examples.The Obj=Subj system discarded dialogues whose subjective and objective assessments disagreed; the other systems used every dialogue.
- 4.2 Dialogue Policy Learning: The on-line GP required only 150 user feedback requests to train a robust reward model, whereas the Obj=Subj and Subj systems requested feedback for every training dialogue.The off-line RNN required no on-line feedback but performed worse than the on-line GP after the first 300 dialogues.
- 4.2 Dialogue Policy Learning: After extending training to 850 dialogues, the on-line GP system performed significantly better than the Subj system, whose user feedback was unreliable.The reported explanation is that the GP noise model mitigated inconsistent feedback; both systems were broadly flat over the longer training span.
- 4.4 Reward Model Evaluation: The GP reward model predicted successful dialogues accurately, although label imbalance lowered recall for failed-dialogue prediction.The imbalance arose because the policy improved during training, producing more positive labels.
5 Conclusion
The paper presents active reward learning with Gaussian process classification and unsupervised dialogue embeddings for on-line policy optimisation. It reports stable optimisation with noisy feedback and better performance in the Cambridge restaurant domain, while acknowledging that the reward focuses mainly on task success.
- Gaussian process classification and an unsupervised neural dialogue embedding enable on-line policy learning in spoken dialogue systems.The embedding requires no labelled training data and provides a compact input to the reward predictor.
- The model robustly handles noisy real-user feedback and uses active learning to minimise feedback requests.Its Bayesian uncertainty estimate supports both active learning and noise handling.
- The proposed approach achieved efficient policy learning and better performance than other state-of-the-art methods in the Cambridge restaurant domain.
- The techniques enable on-line learning in deployed real-world dialogue systems without a large manually annotated corpus or a user simulator.
- The reward function focuses primarily on task success, which may be too simplistic for many commercial applications.The authors identify incorporating additional dialogue-quality dimensions as future work needed for higher user satisfaction.