Source-linked AI summary
DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder
Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, Sunghun Kim
TL;DR
VAE conversation models often match latent posteriors to simple priors, limiting responses to a relatively simple scope. DialogWAE trains a GAN in latent space and uses a Gaussian mixture prior; on two datasets, it outperforms state-of-the-art VAE models and generates more coherent, informative, and diverse responses.
Problem
VAE conversation models match approximate latent posteriors to simple priors, which can restrict generated responses to a relatively simple scope.
Source: §1 · p. 1Method
DialogWAE transforms context-dependent random noise into prior and posterior latent samples, minimizes their Wasserstein distance, and uses a Gaussian mixture prior network.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · pp. 4–5Results
On two widely used datasets, DialogWAE outperforms state-of-the-art VAE models and generates more coherent, informative, and diverse responses.
Source: §6 · p. 9Takeaways & Limitations
The authors report that the GAN architecture with a Gaussian mixture prior facilitates more diverse and informative responses.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Variational autoencoders~(VAEs) have shown a promise in data-driven conversation modeling. However, most VAE conversation models match the approximate posterior distribution over the latent variables to a simple prior such as standard normal distribution, thereby restricting the generated responses to a relatively simple (e.g., unimodal) scope. In this paper, we propose DialogWAE, a conditional Wasserstein autoencoder~(WAE) specially designed for dialogue modeling. Unlike VAEs that impose a simple distribution over the latent variables, DialogWAE models the distribution of data by training a GAN within the latent variable space. Specifically, our model samples from the prior and posterior distributions over the latent variables by transforming context-dependent random noise using neural networks and minimizes the Wasserstein distance between the two distributions. We further develop a Gaussian mixture prior network to enrich the latent space. Experiments on two popular datasets show that DialogWAE outperforms the state-of-the-art approaches in generating more coherent, informative and diverse responses.
1 INTRODUCTION
Dialogue models can struggle to generate meaningful, diverse, on-topic responses, while VAE models may be limited by simple latent priors. DialogWAE trains a GAN in latent space with a Gaussian mixture prior and outperforms state-of-the-art methods on two benchmark datasets.
1 INTRODUCTION
- VAE conversation models can suffer posterior collapse and match latent posteriors to simple priors, restricting responses to a relatively simple scope.The introduction also notes that sequence-to-sequence models often produce safe responses that lack meaning, diversity, or topicality.Source: §1 · p. 1
- Prior GAN approaches face difficulties with discrete-token adversarial training, unstable REINFORCE gradients, or variability limited to the word level.These limitations motivate modeling response distributions in latent space rather than directly over tokens.Source: §1 · p. 1
- DialogWAE transforms context-dependent noise into prior and posterior latent samples and minimizes their Wasserstein distance.A Gaussian mixture prior network is used to model multimodal response variability.Source: §1 · p. 2
- The paper presents DialogWAE as a GAN-based dialogue model that generates latent samples and uses a Gaussian mixture prior to sample from a multimodal distribution.The authors describe it as the first GAN conversation model to exploit multimodal latent structures, to the best of their knowledge.Source: §1 · p. 2
- DialogWAE substantially outperforms state-of-the-art methods on BLEU, word embedding similarity, and distinct across SwitchBoard and DailyDialog.The authors also report that its Gaussian mixture prior facilitates more diverse and informative responses.Source: §1 · p. 2
2 RELATED WORK
Related work addresses safe responses through encoder-decoder variants, dialogue variability through VAE models, and the challenges of applying GANs to discrete language. DialogWAE is distinguished by shaping response distributions in a high-level latent space.
Encoder-decoder variants
- Encoder-decoder variants promote diversity or incorporate situations and topics, whereas this work does not rely on such extra information [1, 7].Li et al. (2015) proposed a diversity-promoting objective; Sato et al. (2017) and Xing et al. (2017) incorporate situations or topic information.Source: §2 · p. 2
VAE conversation models
- VAE conversation models include hierarchical latent-variable approaches and methods addressing posterior collapse with auxiliary losses, dialogue information, or hierarchical structure.DialogWAE addresses VAE limitations using a GAN architecture in latent space.Source: §2 · p. 2
GAN conversation models
- GAN dialogue models have addressed discrete-token nondifferentiability with reinforcement learning or approximate word embeddings, but these approaches have training or variability limitations.DialogWAE instead shapes response distributions in high-level latent space and does not rely on reinforcement learning.Source: §2 · p. 2
- A multimodal distribution is a continuous probability distribution with two or more modes.Source: §2 · p. 2
3 PROPOSED APPROACH
DialogWAE represents response generation with context-conditioned latent variables, then matches learned prior and posterior distributions in latent space. It adds a Gaussian-mixture prior and alternates reconstruction and adversarial training.
3.1 PROBLEM STATEMENT
- The model estimates the next-response distribution pθ(x|c), using latent z to represent the response and an approximate posterior qφ(z|x,c) for inference.The prior pθ(z|c) is conditioned on dialogue context.Source: §3.1 · p. 3
3.2 CONDITIONAL WASSERSTEIN AUTO-ENCODERS FOR DIALOGUE MODELING
- DialogWAE trains a GAN in latent space to match context-dependent prior and posterior samples, minimizing their Wasserstein distance alongside response reconstruction.The prior generator G and posterior generator Q transform context-dependent random noise into latent samples.Source: §3.2 · p. 3
- During training, recognition infers posterior samples from context and response; at generation, the prior network supplies latent samples that the decoder turns into responses.The context encoder forms its representation from utterance encodings and speaker-turn indicators.Source: §3.2 · p. 4
3.3 MULTIMODAL RESPONSE GENERATION WITH A GAUSSIAN MIXTURE PRIOR NETWORK
- The Gaussian mixture prior gives latent noise multiple modes, selecting a component by its mixture probability before sampling within that Gaussian component.The model uses Gumbel-Softmax reparameterization to sample the component indicator.Source: §3.3 · pp. 4–5
3.4 TRAINING
- Training alternates an autoencoder phase minimizing reconstruction loss and a GAN phase minimizing the Wasserstein distance between prior and approximate-posterior latent distributions.Algorithm 1 specifies gradient penalty for discriminator updates and K=3, ncritic=5 in all experiments.Source: §3.4 · p. 5, §3.3 · p. 5
4 EXPERIMENTAL SETUP
The experiments evaluate DialogWAE on DailyDialog and Switchboard using multiple response-quality and diversity metrics against seven dialogue-modeling baselines. The section also specifies the model configuration, training procedure, and test protocol.
Datasets
- DailyDialog and Switchboard are evaluated using the dataset splits adopted by their baseline papers.Switchboard contains 2,400 two-way telephone conversations across 70 topics; the reported splits are 2,316:60:62 for Switchboard and 10:1:1 for DailyDialog.Source: §4 · pp. 5–6
Metrics
- BLEU, BOW embedding similarity, and distinct assess response overlap, embedding similarity, and diversity across sampled responses.Ten responses are sampled per context; BLEU n-gram precision is the average score and recall is the maximum, while BOW reports the maximum embedding score and distinct measures intra- and inter-response diversity.Source: §4 · p. 6, Table 1 · p. 7, Table 2 · p. 7
Baselines
- DialogWAE is compared with seven baselines: HRED, SeqGAN, CVAE, CVAE-BOW, CVAE-CO, VHRED, and VHCR.The performance tables cover Switchboard and DailyDialog.Source: §4 · p. 6, Table 1 · p. 7, Table 2 · p. 7
Training and Evaluation Details
- The model uses 300-unit GRU encoders and decoder, 200-dimensional latent variables, and feed-forward prior, recognition, generator, and discriminator networks.The utterance encoder is bidirectional; the prior and recognition networks have two layers, while the generators and discriminator have three.Source: §4 · p. 6
- Training uses 32-example minibatches, with SGD in the autoencoder phase and RMSprop in the GAN phase.The learning rate decays during autoencoder training; GAN-phase rates are 5×10^-5 for the generator and 1×10^-5 for the discriminator, and evaluation uses the test set after validation tuning.Source: §4 · pp. 6–7
5 EXPERIMENTAL RESULTS
Across two quantitative and qualitative analyses, DialogWAE shows stronger response relevance and diversity than baselines, while its mixture-prior results vary with component count and distinguishable Gaussian components produce different response types.
5.1 QUANTITATIVE ANALYSIS
- Average recall is 42.0% and 37.2% for DialogWAE, versus 29.9% and 26.5% for CVAE baselines across the two datasets.The passage describes these as BLEU-related results and notes a similar trend for BOW embedding metrics.Source: §5.1 · p. 7
- DialogWAE has higher inter-distinct scores than baselines, but no better intra-distinct scores; the authors conjecture that longer responses explain the latter result.The paper links longer responses to fewer repeated n-grams being likely in short responses.Source: §5.1 · p. 7
- With prior-component count K varied from 1 to 9, performance generally rises then falls; the optimum is around 3 on both datasets.The authors attribute the decline to mixture-density training difficulty and insufficient regularization, leaving this for future work.Source: §5.1 · p. 7, Figure 2 · p. 8
5.2 QUALITATIVE ANALYSIS
- In DailyDialog examples, DialogWAE responses cover multiple plausible aspects, while baseline responses show limited variation and often share a prefix such as “how much”.Table 3 shows three generated samples per model for each test context.Source: §5.2 · pp. 7–8, Table 3 · p. 8
- For a dinner-invitation context, the first three Gaussian components produce responses expressing strong willingness, uncertainty, and strong negativity, respectively.The reported overlap between components is around 1/5; the authors interpret this as evidence that the mixture prior captures multimodal response distributions.Source: §5.2 · p. 8, Table 4 · p. 8
- In human evaluation, the proposed approach outperforms CVAE-CO and VHCR by a large margin on coherence, diversity, and informativeness, especially with the Gaussian mixture prior.The evaluation used 50 DailyDialog test dialogues, with responses judged by five participants per context.Source: §5.2 · p. 9, Table 5 · p. 9
6 CONCLUSION
DialogWAE uses a Wasserstein-based latent-variable approach with a Gaussian mixture prior, and experiments on two datasets show stronger response quality than state-of-the-art VAE models.
6 CONCLUSION
- DialogWAE transforms context-dependent Gaussian noise with neural networks to sample prior and posterior latent variables, then minimizes their Wasserstein distance.Source: §6 · p. 9
- A Gaussian mixture prior network enriches the latent space beyond the simple prior distributions used by existing VAE models.Source: §6 · p. 9
- On two widely used datasets, DialogWAE outperforms state-of-the-art VAE models and generates more coherent, informative, and diverse responses.Source: §6 · p. 9