Source-linked AI summary
Latent Intention Dialogue Models
Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, Steve Young
TL;DR
Goal-oriented dialogue models need to represent the decisions underlying responses, while deterministic sequence models struggle to capture natural conversational variability. LIDM learns discrete dialogue intentions through neural variational inference and refines its strategy with reinforcement learning; the paper reports that it outperforms published results in corpus-based and human evaluation.
Problem
Deterministic sequence models struggle to represent underlying dialogue actions and natural conversational variability, while limited goal-oriented dialogue data can lead to overfitting.
Source: §1 · p. 1Method
LIDM learns discrete latent dialogue intentions through neural variational inference and uses them to guide responses, with reinforcement learning available to refine its policy.
Source: §1 · p. 1Results
Takeaways & Limitations
The authors present the discrete latent interface as a promising step toward autonomous dialogue agents that can learn using different paradigms.
Source: §6 · p. 8Later fixed user-side behaviors may not fit the system’s planned scenario, so it provides as much information as possible in early turns.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Developing a dialogue agent that is capable of making autonomous decisions and communicating by natural language is one of the long-term goals of machine learning research. Traditional approaches either rely on hand-crafting a small state-action set for applying reinforcement learning that is not scalable or constructing deterministic models for learning dialogue sentences that fail to capture natural conversational variability. In this paper, we propose a Latent Intention Dialogue Model (LIDM) that employs a discrete latent variable to learn underlying dialogue intentions in the framework of neural variational inference. In a goal-oriented dialogue scenario, these latent intentions can be interpreted as actions guiding the generation of machine responses, which can be further refined autonomously by reinforcement learning. The experimental evaluation of LIDM shows that the model out-performs published benchmarks for both corpus-based and human evaluation, demonstrating the effectiveness of discrete latent variable models for learning goal-oriented dialogues.
1. Introduction
LIDM models dialogue intentions as discrete latent variables that guide responses and can be refined through reinforcement learning. The paper reports state-of-the-art performance on corpus-based and human evaluation.
1. Introduction
- Sequence-to-sequence models learn conditional string outputs but miss underlying dialogue actions, limiting diverse responses and risking overfitting when goal-oriented dialogue data are insufficient.Source: §1 · p. 1
- Discrete latent variables offer interpretable distributions and a semi-supervised learning framework, though sampling has high variance that REINFORCE-style baselines can mitigate.Source: §1 · p. 1
- LIDM infers intentions from user utterances and dialogue context, learning from a variational lower bound and from clustered intention labels in a labeled corpus subset.Source: §1 · p. 2
- The learned intention distribution acts as an initial policy that policy-gradient reinforcement learning can refine against alternative objectives.Source: §1 · p. 2
- LIDM achieves state-of-the-art performance on automatic corpus-based and human evaluation.Source: §1 · p. 2
2. Latent Intention Dialogue Model for Goal-oriented Dialogue
LIDM represents goal-oriented dialogue decisions with a discrete latent intention that guides response generation, using dialogue state built from the user utterance, belief state, and knowledge-base matches. Variational inference, semi-supervised training, and reinforcement learning provide ways to learn and refine this policy.
2. Latent Intention Dialogue Model for Goal-oriented Dialogue
- The goal-oriented setting focuses on information-seeking dialogues, where a system parses user input into commands, searches a knowledge base, and replies in natural language.The stated goal is to help users complete tasks through natural-language interaction.Source: §2 · p. 2
2.1. Model
- A dialogue state combines the encoded user utterance, belief vector, and knowledge-base match vector; a sampled latent intention then controls response generation.The policy network conditions the intention on the dialogue state, and the sampled intention is combined with that state to guide a conditional LSTM language model.Source: §2.1 · pp. 2–3
2.2. Inference
- The inference network approximates the intention posterior for variational training, while the policy network supplies intention samples for generating responses.The REINFORCE estimator has high variance; the method introduces a constant and a state-dependent baseline to reduce it.Source: §2.2 · pp. 3–4
2.3. Semi-Supervision
- Semi-supervised training uses clustering to assign intention labels to some responses, then combines labeled-example likelihood with the unlabeled variational objective.The trade-off parameter α controls the balance between supervised and unsupervised examples.Source: §2.3 · p. 4
2.4. Reinforcement Learning
- Reinforcement learning fine-tunes the learned policy against alternative objectives by updating policy-network parameters, while labeled examples retain their assigned actions.Updating only the policy parameters is described as refining decision-making while avoiding over-fitting.Source: §2.4 · pp. 4–5
3. Experiments
The experiments evaluate LIDM on CamRest676 using corpus-based and human judgments. RL fine-tuning improves corpus task success over published baselines without significantly sacrificing BLEU, while human ratings favor LIDM models on comprehension and naturalness.
3.1. Dataset & Setup for Goal-oriented Dialogue
- CamRest676 contains 676 restaurant-search dialogues and about 2,750 turns; the experiments use a 3:1:1 train-validation-test split.The task concerns finding restaurants in Cambridge, UK.Source: §3.1 · p. 5
- Responses were clustered by content words to create semi-supervised intention seeds, labeling about 35% to 43% of responses across the dataset.The share varied with the number of latent dimensions, from 35% at I = 50 to 43% at I = 100.Source: §3.1 · p. 5
3.2. Experiments on Goal-oriented Dialogue
- LIDM+RL outperformed three published baselines on task success without significantly sacrificing BLEU.The corpus-based evaluation compares LIDM with and without RL against the three published models.Source: §3.2 · p. 6, Table 2 · p. 6
- Human judges rated LIDM’s subjective success almost indistinguishably from the other models, while LIDM models outperformed NDM on comprehension and naturalness.The evaluation collected 200 dialogues per model and rated success, comprehension, and naturalness on a 1-to-5 scale.Source: §3.2 · p. 6, Table 3 · p. 6
- The corpus-based success metric favors greedy policies because user behavior is fixed, unlike an interactional conversation.The system may provide information early in anticipation of later user behavior that does not fit its planned scenario.Source: §3.2 · p. 6
- Example dialogues show LIDM’s varied intention modes, while LIDM+RL acts greedily, sometimes providing an address and phone number before they are requested.LIDM+RL dialogues were shorter on average than those of the other evaluated models.Source: §3.2 · pp. 6–7, Table 4 · p. 7
4. Discussion
The latent variable separates dialogue language generation from internal decision-making, addressing credit assignment by allowing different learning signals to update different submodules. It also links variational and reinforcement learning through a shared intention interface.
4. Discussion
- The discrete latent variable decomposes language learning from dialogue decision-making, helping address credit assignment through separate learning signals for different submodules.The latent distribution is updated by the variational lower-bound reward or by dialogue-success and sentence-BLEU rewards in reinforcement learning.Source: §4 · p. 7
- The latent variable bridges Bayesian learning and reinforcement learning by bringing their distinct update signals together within one framework.The paper characterizes the variational lower bound and dialogue success and sentence BLEU as the respective reward signals.Source: §4 · p. 7
- The framework explicitly models hidden dialogue intentions and lets the agent learn a dialogue policy through interaction.Source: §4 · p. 7
5. Related work
LIDM combines stochastic response generation with explicit modeling of hidden intentions, contrasting with approaches based on fixed handcrafted intentions or deterministic end-to-end models. Its discrete intention space can also receive signals from different learning paradigms.
5. Related work
- LIDM uses latent variables both to add stochasticity to responses and to model hidden dialogue intentions explicitly, combining generative modeling with reinforcement learning.Source: §5 · p. 7
- The LIDM+RL dialogue examples show earlier provision of the address and phone number than the user requests them.Source: Table 5 · p. 8
- POMDP-based systems use modular pipelines and fixed handcrafted dialogue acts, while LIDM infers intentions from data and handles long-tailed intention distributions through similarity measurement.Source: §5 · pp. 7–8
- End-to-end goal-oriented models are described as typically deterministic and reliant on decoder supervision to fine-tune many parameters.Source: §5 · p. 8
- LIDM’s discrete intention space can bootstrap from signals across paradigms, including semi-supervised learning and reinforcement learning.Source: §5 · p. 8
6. Conclusion
The conclusion presents LIDM as a discrete-latent framework for goal-oriented dialogue that learns an initial policy and can revise it using external rewards. The experiments showed communication with human subjects and performance above previously published results.
6. Conclusion
- LIDM learns an initial policy from dialogue data and can revise its strategy using external rewards through reinforcement learning.Source: §6 · p. 8
- Experiments showed that LIDM communicated with human subjects and outperformed previous published results.Source: §6 · p. 8
- The authors describe the learned discrete latent interface as a promising step toward autonomous agents that learn using different paradigms.Source: §6 · p. 8