Source-linked AI summary
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
Tiancheng Zhao, Kaige Xie, Maxine Eskenazi
TL;DR
Defining action spaces for dialog RL remains challenging: handcrafted actions and word-level actions each have limitations. The paper learns latent actions for end-to-end dialog agents and evaluates them on DealOrNoDeal and MultiWoz. It reports stronger results than word-level RL and identifies discrete latent actions as more suitable than continuous ones.
Problem
Dialog RL lacks an action-space choice that avoids the limitations of handcrafted representations and word-level output actions.
Source: §1 · p. 1Method
Results
LaRL outperforms word-level RL on DealOrNoDeal and MultiWoz without incomprehensible generation, with an 18.2% absolute improvement over prior state of the art on MultiWoz.
Source: §1 · p. 2Takeaways & Limitations
The analysis finds discrete latent actions more suitable than continuous ones for dialog RL and identifies reducing exposure bias as important.
Source: §1 · p. 2, §8 · p. 9The baseline RL setup assumes an interactive environment that provides turn-level rewards at every dialog turn.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have their own limitations. This paper proposes a novel latent action framework that treats the action spaces of an end-to-end dialog agent as latent variables and develops unsupervised methods in order to induce its own action space from the data. Comprehensive experiments are conducted examining both continuous and discrete action types and two different optimization methods based on stochastic variational inference. Results show that the proposed latent actions achieve superior empirical performance improvement over previous word-level policy gradient methods on both DealOrNoDeal and MultiWoz dialogs. Our detailed analysis also provides insights about various latent variable approaches for policy learning and can serve as a foundation for developing better latent actions in future research.
1 Introduction
The introduction frames action-space design as a barrier to reinforcement learning in end-to-end dialog and proposes learning discourse-level actions from data. It reports gains over word-level reinforcement learning on two dialog tasks, alongside analysis of latent-action training choices.
1 Introduction
- Handcrafted dialog acts restrict modular systems to domains whose action spaces can be represented by those designs, limiting complex-domain conversations.The cited discussion associates these limits with prior work.Source: §1 · p. 1
- Word-level RL treats every response word as an action, but can produce incomprehensible language and faces long-horizon credit assignment that slows and weakens convergence.A dialog may span hundreds of words, according to the introduction.Source: §1 · p. 1
- LaRL induces expressive discourse-level latent actions from conversational data, separating dialog decisions from natural-language generation.The paper also proposes a training objective beyond the typical evidence lower bound and attention for integrating discrete variables in long responses.Source: §1 · pp. 1–2
- LaRL improves over word-level RL on DealOrNoDeal and MultiWoz, avoids incomprehensible generation, and reports an 18.2% absolute improvement over prior state of the art on MultiWoz.Its analyses also find that discrete latent actions are more suitable than continuous ones and that reducing exposure bias is important.Source: §1 · p. 2
2 Related Work
Related work covers reinforcement learning in modular and end-to-end dialog systems, as well as latent-variable models. The paper distinguishes its approach by learning expressive latent actions unsupervised for policy optimization.
2 Related Work
- Prior RL research in modular dialog management optimizes policies over hand-crafted action spaces in task-oriented domains; managers are modeled as POMDPs, with dialog states estimated by tracking models, and RL finds optimal policies.Source: §2 · p. 2
- End-to-end RL has been applied to open-domain chat, negotiation, visual, and grounded dialog, but typically uses each output word as an action and can harm language quality and convergence (Serban et al., 2017a; Mordatch and Abbeel, 2017).The discussion identifies deviation from natural language and suboptimal convergence as limitations.Source: §2 · p. 2
- Earlier latent-variable dialog work reports benefits including diverse decoding, interpretable decisions, and zero-shot transfer, while some coarse nodes decouple generation from policy (Serban et al., 2017b; Zhao et al., 2017; Cao and Clark, 2017; Wen et al., 2017; Zhao et al., 2018; He et al., 2018; Yarats and Lewis, 2017).The paper says those prior latent actions were auxiliary, small-scale, and mostly supervised or semi-supervised; it presents its work as a comprehensive study of unsupervised latent variables for RL policy optimization.Source: §2 · p. 2
3 Baseline Approach
The baseline models conditional response generation from dialog context and trains with supervised learning followed by policy-gradient RL. Its word-level policy treats every output token as an action, creating a large action space and long learning horizon.
3 Baseline Approach
- E2E response generation is treated as conditional language generation using neural encoder-decoders (Cho et al., 2014) to model p(x|c), where c is the observed dialog context and x is the system’s response.Figure 1 compares word-level and latent-action reinforcement learning in a sample multi-turn dialog; green boxes denote the decoder generating a response from latent code z, and dashed lines mark where task-reward policy gradients are applied.Source: §3 · p. 2, Figure 1 · p. 3
- The baseline first maximizes log likelihood on training dialogs, then uses policy gradients such as REINFORCE to optimize task-dependent goals.The RL setup assumes an interactive environment and a turn-level reward at each dialog turn.Source: §3 · p. 3
- Word-level RL treats every output word as an action, producing an action space of size |V| and a learning horizon of TU.The passage says direct application can make the decoder diverge; alternating policy-gradient and supervised updates is a common response.Source: §3 · p. 3
- The RL:SL=A:B ratio specifies B supervised updates for every A policy-gradient updates; “off” means no supervised updates.The paper introduces this schedule to describe alternation with supervised learning.Source: §3 · p. 3
4 Latent Action Reinforcement Learning
LaRL replaces word-level actions with context-conditioned latent actions that determine response generation, separating discourse-level decisions from surface wording. The paper compares Gaussian and categorical actions and proposes a simplified training objective to address latent-space exposure bias.
4 Latent Action Reinforcement Learning
- LaRL samples a context-conditioned latent action z and generates the response from z, making the latent variable—not each output word—the action space.The encoder chooses z from dialog context, while the decoder maps z to surface-form words.Source: §4 · p. 3
- LaRL shortens the policy-gradient horizon to T and updates the encoder while leaving the decoder intact, separating action choice from word generation.The latent action space is designed to be low-dimensional relative to the output vocabulary.Source: §4 · p. 3
4.1 Types of Latent Actions
- Gaussian actions use an M-dimensional diagonal-covariance distribution whose context encoder projects dialog context into the distribution’s mean and variance.The sampled z initializes the response decoder.Source: §4.1 · p. 4
- Categorical actions use M independent K-way variables, allowing K^M combinations; attention fusion lets the decoder attend to different latent dimensions during generation.The proposed fusion addresses summation fusion’s loss of fine-grained order information and difficulty with long responses involving multiple dialog acts.Source: §4.1 · p. 4
4.2 Optimization Approaches
- Full ELBO training approximates the posterior with q(z|x,c), while the proposed Lite ELBO sets that posterior equal to the context encoder and adds prior regularization.The added KL regularizer addresses overfitting from an otherwise unregularized latent space; the paper uses a uniform categorical prior or N(0, I) Gaussian prior.Source: §4.2 · pp. 4–5
5 Experiment Settings
The experiments evaluate LaRL on negotiation and task-oriented dialog settings, including a newly constructed Multi-Woz reinforcement-learning task. They also use Language Constrained Reward curves to compare task reward while accounting for language fidelity.
5.1 DealOrNoDeal Corpus and RL Setup
- DealOrNoDeal evaluation uses 5,805 negotiation dialogs from 2,236 scenarios and measures perplexity, reward, agreement, and diversity with shared judge and simulator models.Perplexity tracks human-likeness; reward and agreement assess negotiation strength, while diversity indicates whether novel discourse-level strategies emerge.Source: §5.1 · p. 5
5.2 Multi-Woz Corpus and Novel RL Setup
- Multi-Woz evaluation uses the Dialog-Context-to-Text task with ground-truth belief states and measures BLEU, Inform Rate, and Success Rate.The RL episode samples a training dialog, runs the model on every system turn without changing the original context, computes Success Rate, and updates policy-gradient parameters; the authors characterize this as similar to contextual bandits and report that word-level RL falls short.Source: §5.2 · p. 5
- The Multi-Woz RL setting is challenging, and word-level RL falls short.The authors describe the task as a variant similar to contextual bandits, with the goal of generating responses that yield better Success Rate.Source: §5.2 · p. 5
5.3 Language Constrained Reward (LCR) curve for Evaluation
- The Language Constrained Reward curve plots the best test reward achievable under a maximum-perplexity budget, exposing the tradeoff between reward and language fidelity.A perfect model should lie in the upper-left corner; models sacrificing language quality for reward lie toward the lower-right.Source: §5.3 · pp. 5–6
6 Results: Latent Actions or Words?
The section compares the best latent-action and word-level RL models on DealOrNoDeal and MultiWoz. The results show task-specific gains for latent actions alongside smaller losses in language quality and greater response diversity.
6 Results: Latent Actions or Words?
- The validation-selected comparison uses baseline HRED, tuned to reproduce results from, fine-tuned with word-level RL at RL:SL=4:1, versus best-performing LiteCat.During RL training, RL:SL=off for all latent-action models, while baseline word-level RL models can tune RL:SL for best performance.Source: §6.1 · p. 6, §6 · p. 6
6.1 DealOrNoDeal
- LaRL models achieve strong task rewards with a smaller PPL drop, while word-level RL raises rewards by deviating significantly from natural language.The comparison is shown in the DealOrNoDeal LCR curves.Source: Figure 2 · p. 6, §6.1 · p. 6
- After RL training, word-level baseline diversity falls to 5 unique responses, while LiteCat diversity rises from 58 to 202.The baseline repeatedly uses “hat” and “ball”; LiteCat uses varied negotiation strategies.Source: §6.1 · p. 6, Figure 3 · p. 7
6.2 MultiWoz
- 18.24% absolute improvement over the MLE-based state-of-the-art: LiteAttnCat nears human success and inform rates as PPL rises from 4.05 to 5.22.Word RL reaches 79% success, but its responses depart from natural language.Source: Table 3 · p. 7, §6.2 · p. 7
7 Model Analysis
The analysis compares latent-action model variants and finds that the Lite objective, regularization, and categorical actions each have advantages over alternatives. It also distinguishes diversity learned during reward optimization from diversity during pre-training.
7 Model Analysis
- Figure 5 plots LCR curves for DealOrNoDeal and MultiWoz, excluding Lfull models because their PPLs are too poor for comparison with Lite models.The curves cover the three models pre-trained with Llite and fine-tuned using policy-gradient reinforcement learning.Source: Figure 5 · p. 8, §7 · p. 8
- Lite pre-training outperforms Lfull on PPL and BLEU, while Lfull has higher pre-training diversity.The authors attribute the weaker Lfull metrics to latent-space exposure bias; Lite models can increase diversity during RL to improve negotiation reward.Source: §7 · p. 8, Table 5 · p. 8
- With β = 0, LiteCat and LiteGauss reach lower final rewards than their regularized versions with β = 0.01.The authors attribute this to low entropy in the unregularized pre-trained policy, which limits exploration during RL.Source: §7 · p. 8, Table 6 · p. 8
- Categorical latent actions consistently outperform Gaussian actions; REINFORCE with Gaussian actions is unstable and can cause divergence.The authors suspect exploration in the unbounded Gaussian latent space can reach regions not covered during supervised training, where decoder behavior is undefined.Source: §7 · p. 8
8 Conclusion and Future Work
The paper proposes a latent-variable action space for reinforcement learning in end-to-end dialog agents. It reports superior performance and a new state-of-the-art MultiWoz success rate, alongside analyses of latent-variable training.
8 Conclusion and Future Work
- The proposed framework uses a latent-variable action space for reinforcement learning in end-to-end dialog agents.It includes a regularized ELBO objective and attention fusion for discrete variables, and is assessed on two dialog tasks using the proposed LCR curve.Source: §8 · p. 9
- The models achieve superior performance and a new state-of-the-art success rate on MultiWoz.The paper presents its analyses as insights into training latent variables as dialog-agent action spaces and describes the work as a first step in this research direction.Source: §8 · p. 9
- The approach creates action abstraction for dialog agents in an unsupervised manner.The authors situate this work within research on practical latent variables in dialog agents.Source: §8 · p. 9
A.1 Training Details
The appendix identifies separate training-detail tables and hyperparameter descriptions for DealOrNoDeal and MultiWoz.
A.1 Training Details
- DealOrNoDeal results have a stated hyperparameter set and a corresponding training-details table.The table caption identifies Attn GRU as referring to Yang et al. (2016).Source: §A.1 · p. 11, Table 7 · p. 11
- The appendix organizes the training information by dataset rather than combining both tasks in one table.DealOrNoDeal and MultiWoz each have their own training-details table.Source: Table 7 · p. 11, Table 8 · p. 11
- MultiWoz results have a stated hyperparameter set and a separate training-details table.The appendix distinguishes these details from those used for DealOrNoDeal.Source: §A.1 · p. 11, Table 8 · p. 11
A.2 DealOrNoDeal Example Conversations
The appendix provides example conversations from word-level and latent-level policy-gradient models.
A.2 DealOrNoDeal Example Conversations
- The appendix presents example conversations generated by word-level and latent-level models.The word-level example uses an agent trained with word-level policy gradient against a supervised pre-trained user model; the latent-level example uses LiteCat against that user model.Source: §A.2 · p. 11, Table 9 · p. 12, Table 10 · p. 12
- Table 9 shows a baseline agent trained with word-level policy gradient interacting with a supervised pre-trained user model.The appendix identifies this as an example dialog.Source: Table 9 · p. 12
- Table 10 shows LiteCat trained with latent-level policy gradient interacting with a supervised pre-trained user model.The appendix identifies this as an example dialog.Source: Table 10 · p. 12