A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

Iulian Vlad Serban; Alessandro Sordoni; Ryan Lowe; Laurent Charlin; Joelle Pineau; Aaron Courville; Yoshua Bengio

Source-linked AI summary

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, Yoshua Bengio

arXiv:1605.06069v3May 19, 2016cs.CLcs.AIcs.LGcs.NE

TL;DR

Dialogue generation requires modeling both local utterance structure and higher-level dependencies across utterances, which conventional RNN-based models represent weakly. The paper introduces VHRED, a hierarchical latent-variable encoder-decoder trained variationally and evaluates it with automatic and human measures. VHRED improves over recent models, with latent variables associated with longer, more informative outputs and better context maintenance.

Problem
Hierarchical dialogue structure combines local word-level statistics with higher-level uncertainty across utterances, while RNNLM and HRED place variation only in token outputs.
Method
VHRED augments HRED with latent variables sampled before token generation for each hierarchical subsequence, maintaining long-term context and using a variational training objective.
Results
VHRED improves upon recently proposed models in dialogue response generation, including response quality in human evaluation and topic tracking across generated turns.
Takeaways & Limitations
The results highlight hierarchical generation as advantageous for modeling high-entropy sequences, while latent variables support longer, more informative responses and dialogue-context maintenance.

Abstract

from arXiv · show

Sequential data often possesses a hierarchical structure with complex dependencies between subsequences, such as found between the utterances in a dialogue. In an effort to model this kind of generative process, we propose a neural network-based generative architecture, with latent stochastic variables that span a variable number of time steps. We apply the proposed model to the task of dialogue response generation and compare it with recent neural network architectures. We evaluate the model performance through automatic evaluation metrics and by carrying out a human evaluation. The experiments demonstrate that our model improves upon recently proposed models and that the latent variables facilitate the generation of long outputs and maintain the context.

1 Introduction

The paper motivates hierarchical stochastic modeling for dialogue because conventional RNNs inject variability only at token output, missing higher-level uncertainty across utterances. It introduces VHRED and evaluates it against recent architectures using automatic metrics and human judgments.

Dialogue has at least two structural levels: local word-level statistics within utterances and uncertainty across utterances involving topic, goals, and style.
Conventional RNNs place their only model variability in the sampled output, which is inappropriate for hierarchical sequential data.
The proposed architecture uses hierarchical stochastic latent variables to model generative processes with multiple levels of variability.
VHRED is evaluated for dialogue response generation against recent neural architectures using manual inspection, human evaluation, and automatic metrics.
The results report improved performance over recently proposed models, with latent variables supporting longer, more informative utterances and dialogue-context maintenance.

2 Technical Background

The technical background describes RNN language modeling and HRED as deterministic hierarchical extensions, then identifies their tendency to prioritize short-term token prediction over long-term dialogue structure.

2.1 Recurrent Neural Network Language Model: An RNN recursively updates a hidden state from each observed token and uses that state to parameterize the next-token output distribution.
2.1 Recurrent Neural Network Language Model: RNN language-model parameters are learned by maximizing training log-likelihood with gradient descent.
2.2 Hierarchical Recurrent Encoder-Decoder: HRED models dialogues as utterance sequences and words within utterances using encoder, context, and decoder RNN modules.
2.3 A Deficient Generation Process: RNNLM and HRED constrain variation to the conditional token-output distribution, encouraging local rather than global or long-term structure.
2.3 A Deficient Generation Process: In high-entropy sequences, vanishing gradients and noisy token-level perturbations make short-term prediction easier than sustaining long-term trajectories.

3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED)

VHRED augments HRED with a stochastic latent variable for each subsequence, separating latent sequence-level sampling from token generation while preserving context. It uses prior and posterior Gaussian distributions and trains latent variables with a variational lower bound.

3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): VHRED introduces a latent variable at the decoder and generates each hierarchical subsequence in two steps: sample the latent variable, then generate tokens.
3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): For each subsequence, VHRED samples z_n conditioned on previous subsequences and then generates the subsequence tokens conditioned on z_n and prior tokens.
3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): The latent prior is a diagonal-covariance multivariate normal distribution whose mean and covariance depend on preceding subsequences.
3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): VHRED retains HRED’s encoder, context, and decoder RNNs, while transforming context representations into the parameters of the latent prior.
3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): During training, latent variables are inferred by maximizing a variational lower bound that combines a reconstruction expectation with a KL-divergence penalty.
3 Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED): The architecture models variation at both sequence level through z and token level through the conditional distribution over words, supporting long-term output trajectories.

4 Experimental Evaluation

The experiments evaluate VHRED for conditional dialogue response generation on Twitter and Ubuntu using human judgments, qualitative inspection, and automatic metrics. VHRED is generally preferred and produces longer, more informative, and more context-relevant responses than baseline models.

Datasets and baselines: Experiments use Twitter and Ubuntu dialogue corpora, comparing VHRED with LSTM, HRED, and TF-IDF retrieval baselines.Twitter dialogues typically involve social interaction, whereas Ubuntu dialogues are goal-driven technical conversations.
Evaluation procedures: The evaluation combines Amazon Mechanical Turk pairwise judgments, qualitative response inspection, and automatic statistics and similarity metrics.Human evaluation was conducted on Twitter, while Ubuntu responses were excluded because technical expertise was uncommon among AMT users.
Human evaluation: VHRED is preferred in most human-evaluation comparisons, outperforming HRED and TF-IDF for both short and long contexts.VHRED also beats LSTM for long contexts, while LSTM is preferred for short contexts.
Human evaluation: Qualitative examples show VHRED generating longer, more meaningful, and more varied responses, including improved handling of smilies, slang, multilingual continuation, and imaginative actions.Some interesting VHRED responses may be less preferred when slightly incompatible with context.
Metric-based evaluation: VHRED better captures ground-truth response topics than LSTM and HRED, especially when generating three consecutive utterances.The authors interpret this pattern as evidence that stochastic latent variables help balance short-term and long-term generation.
Metric-based evaluation: VHRED produces higher per-word entropy on both datasets and, on Twitter, responses containing 6 bits more information on average than HRED.It also produces longer Twitter responses, suggesting greater response diversity than LSTM and HRED.

5 Related Work

The paper situates VHRED among variational sequence models and distinguishes it through hierarchical conditioning, context transfer, and conditional dialogue generation. An initial experiment also found its deterministic context-to-decoder connection beneficial for the variational bound.

Variational sequence models: VHRED builds on variational autoencoders and related sequence models such as VRNN, STORN, Variational Recurrent Autoencoder, and Variational Autoencoder Language Model.These prior architectures were applied to domains including images, speech, handwriting, music, and language.
Distinctive architecture: VHRED conditions each latent variable on all previous sentences, allowing generation of multiple dependent sub-sequences while making latent variables co-dependent through observed tokens.This differs from related encoder-decoder applications in generative music and language modeling.
Distinctive architecture: VHRED combines hierarchical HRED structure with a deterministic context-to-decoder connection, supporting generation conditioned on long contexts.The authors report that initial experiments confirmed this connection lowered the variational bound.
Task distinction: Unlike autoencoder frameworks focused on input reconstruction, VHRED reports improved results for conditional generation of the next dialogue utterance.The comparison emphasizes the dialogue-response objective rather than reconstruction.

6 Discussion

VHRED is a hierarchical latent-variable architecture for sequential generation, demonstrated on dialogue response generation and positioned as broadly applicable to hierarchical sequence tasks.

VHRED uses a hierarchical generation process to exploit structure in sequences and is trained with a variational lower bound on log-likelihood.
The model improves previous models across several measures, including response quality in a human study.
The empirical results highlight hierarchical generation's advantages for modelling high-entropy sequences.
VHRED can in principle be applied to hierarchical tasks including document-level translation, web query prediction, and multi-sentence summarization or image captioning.

Appendix

The appendix describes the dialogue corpora used and notes plans to release preprocessing and model implementations publicly.

The Twitter Dialogue Corpus was extracted in 2011, minimally preprocessed, and tokenized with the Moses tokenizer.
The Ubuntu Dialogue Corpus v2.0 was extracted in January 2016, and its preprocessed version was slated for public release.
The model implementations were planned for public release upon paper acceptance.

Training and Generation

Validation is performed periodically, while test-time responses are generated with length-normalized beam search using five beams.

Each model is validated on the entire validation set every 5000 training batches.
At test time, RNN decoders use beam search with 5 beams to output responses.
Beam cost is defined as beam-token log-likelihood divided by the number of tokens in the beam.
The passage contrasts beam search with sampling, which is described as producing poorer results for these decoders.

Baseline Models

The baseline implementations use feed-forward gating between context and decoder RNNs, with additional decoder-context and encoder-state connections on Twitter.

On Ubuntu, the context-to-decoder RNN gating function is a one-layer feed-forward network with hyperbolic tangent activation.
On Twitter, the HRED decoder produces a 1000-dimensional vector at each hidden step and combines it with the output context RNN.
The combined Twitter representation passes through a tanh feed-forward network before entering the decoder RNN.
For Twitter, each utterance encoder RNN starts from the previous utterance encoder's last hidden state.

Latent Variable Parametrization

The model defines hierarchical latent-variable priors and approximate posteriors from dialogue context and encoder states, sampling one latent variable at each utterance boundary. The Ubuntu examples provide model-generated response illustrations, while Table 5 marks turn changes with a special token.

Prior: The prior distribution uses the HRED context encoder state to compute latent-variable parameters before generation.The context state is transformed into a prior representation, whose mean and covariance parameterize z_t.
Prior: At generation time, the model samples z_t from the prior Gaussian distribution at the end of each utterance.The latent variable is sampled as z_t ∼ N(µ_t,prior, Σ_t,prior).
Approximate posterior: The approximate posterior combines the context state with the encoder state of the next subsequence to compute posterior parameters.The concatenated state h_t,p has dimensionality d_h,con + d_h,enc.
Approximate posterior: The variational-bound derivative is estimated by sampling one posterior latent variable at each utterance boundary.The posterior sample follows z_t ∼ N(µ_t,posterior, Σ_t,posterior).
Model examples: Table 5 presents Ubuntu model examples and uses the → token to indicate a change of turn.The cited material identifies the table as a collection of Ubuntu model examples and defines the turn-marker token.

Setup

The human evaluation uses randomized AMT preference tests in which annotators compare candidate responses for sampled, filtered dialogue contexts. The setup includes indeterminable judgments, attention checks, and multiple independent evaluators per batch.

Evaluation rationale: AMT is used instead of in-lab evaluation to access a larger, more heterogeneous, and more replicable annotator pool.The authors acknowledge that in-lab experiments usually have less noise and higher annotator agreement.
Evaluation protocol: Evaluators may mark either response as indeterminable when unfamiliar topics, slang, or non-English language impede understanding.This option is intended to avoid forcing a preference when the dialogue context is not understood.
Evaluation protocol: The task asks evaluators to choose between two candidate next utterances without telling them that responses may be computer-generated.The setup resembles a restricted Turing Test and imposes only minimum context and response-length requirements.
Quality control: Examples and candidate-response order are shuffled, and four attention checks remove evaluators who fail more than one check.The procedure is designed to reduce presentation bias and screen for task comprehension.
Sampling and filtering: Contexts were sampled from the test set and filtered for non-English and offensive content using automatic screening followed by manual skimming.The passage notes that the automatic filtering was not perfect.
Sampling and scale: 5363 preferences were collected in 69 AMT HITs, with each HIT containing 70–90 examples evaluated by 3–4 unique humans.For each model pair, the authors conducted 3–5 HITs.