Mixture of Decoders for Diverse Dialog Response Generation

1 INTRODUCTION

• Sequence-to-sequence models tend to place their output mode on dull responses despite the many plausible, lexically and syntactically varied replies for a context.The paper describes this degenerate output distribution as “modal collapse.”Source: §1 · p. 1
• Mixture of decoders (MoD) assigns a dedicated decoder to each discrete latent category, aiming to learn specialized components of multi-modal response distributions.The categories can map to semantic or pragmatic groupings such as topics or dialog acts.Source: §1 · p. 1
• The model uses conditional variational autoencoding for learning and inference, and the paper evaluates recurrent-neural-network decoders.The authors conjecture that the approach could also apply to convolutional or Transformer decoders.Source: §1 · pp. 1–2
• The authors present MoD as the first deep-learning application of mixture modelling to dialog response generation and as an extension of the M2 semi-supervised VAE with a context-conditioned prior.Discrete categories also permit incorporating prior topic or dialog-act knowledge through semi-supervised learning.Source: §1 · pp. 1–2

2 RELATED WORK

• Existing diversity approaches modify training or decoding objectives, reweight examples, or iteratively boost training; the paper’s approach augments existing models and can be combined with these methods.The paper surveys these approaches as alternatives to changing the underlying model capacity.Source: §2 · p. 2

2.1 LATENT VARIABLE MODEL

• Dialog models with continuous latent variables condition generation on context, while discrete mixtures permit label-based prior knowledge and greater interpretability.Figure 1 illustrates the degenerate unimodal response distribution attributed to sequence-to-sequence models.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 MIXTURE MODEL FOR LANGUAGE

• For language mixtures, prior work uses uniform or context-agnostic mixture priors, whereas this paper conditions the mixture prior on context; mixture-of-softmax instead mixes output-word distributions at each time step.Figure 2 depicts two mixture-of-decoder components, each a variational auto-decoder, over their first two recurrent steps; attention is omitted.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

3.1 FORMULATION

• MoD adds discrete mixture variable y to a continuous-latent response model, using a variational posterior and maximizing the evidence lower bound.The variational posterior factorizes into distributions over y and z conditioned on the response and context.Source: §3.1 · pp. 3–4

3.2 DECODING

• Decoding greedily uses the continuous latent variable’s prior mean, then nominates candidates with per-mixture beam search and ranks them by weighted mixture scores.The voting scores are weighted by the conditional prior probabilities of the mixtures.Source: §3.2 · p. 4

3.3 ARCHITECTURES

• A bidirectional LSTM encodes dialog history, each mixture has its own LSTM decoder, and response encoders support posterior inference over mixtures and z.The mixture prior uses encoder states; the time-step prior for z uses decoder states and attention context.Source: §3.3 · pp. 4–5

3.4 SEMI-SUPERVISED LEARNING

• Bootstrap labels train the posterior discriminatively: lexical-embedding k-means assigns clusters, while dull responses are assigned to a dedicated mixture.Training still maximizes word likelihood over all mixtures for labeled data, allowing other mixtures to learn those responses with lower weights.Source: §3.4 · p. 5

3.5 MODEL COMPARISON

• MoD combines decoder state, attention context, and continuous latent information through a bottleneck for word prediction, sharing the output-layer weights across mixtures.The bottleneck is used to reduce parameters; its size is usually set to the word-embedding size.Source: §3.5 · p. 6
• MoD uses far fewer parameters than MoS, whose parameter count grows very quickly with the number of mixtures, while MoD learns rich latent representations.MoD uses distinct parameters across multiple layers while keeping parameter counts small.Source: Table 2 · p. 6, §3.5 · p. 7

3.6 IMPLEMENTATION AND TRAINING DETAILS

• Mixture-based models train three to five times more slowly than baseline models without mixtures.The implementation uses 512-unit LSTM encoders and decoders and 300-dimensional word embeddings.Source: §3.6 · p. 7

4.1 DATA

• The experiments use Persona, an open-domain corpus with persona descriptions, and assess response length, distinct n-grams, BLEU, ROUGE-L, and sentence-embedding cosine similarity.Responses use the last utterance as source with persona descriptions prepended; word-averaged sentence embeddings measure similarity to human responses and persona descriptions.Source: §4.1 · p. 7, §4.2.1 · p. 7, §4.2.2 · p. 7

4.2.2 WORD OVERLAP BASED METRICS

• Figure 3 varies the number of mixtures across K = 1, 3, 4, and 5; increasing K may improve lexical diversity but does not always improve reference-word matching, and MoS ROUGE-L declines from K = 3.The maximum K was 5 because of a GPU memory constraint.Source: Figure 3 · p. 8, §4.2.3 · p. 8

4.2.3 WORD EMBEDDING BASED METRICS

• MoD beat baseline models on word-overlap metrics, and adding continuous latent variables improved those metrics; MoS responses most closely matched human lengths, while some MoD responses repeated themselves.VAD tended to generate short responses, whereas MoD tended to generate longer ones; BLEU scores were correlated with length because of the brevity penalty.Source: §4.2.3 · p. 8
• Unsupervised MoD performed best on matching ground-truth responses, supervised MoD performed best on matching persona descriptions, and full MoD also compared well with baselines.Source: §4.2.3 · p. 8
• VAD, MoS, and MoD all exceeded the sequence-to-sequence baseline in lexical diversity; continuous latent variables improved diversity, and supervision further improved MoD.Source: §4.2.3 · p. 8

4.3 QUALITATIVE ANALYSIS

• In paired-response judgments on appropriateness and persona relevance, annotators preferred MoD over VAD and MoS more than 40% of the time, with losses under 30% and ties about 30%.Source: §4.3 · p. 9
• Examples suggest the five supervised MoD mixtures specialize in different topics, including hobbies, education, conversation starters, food, and occupation or leisure.Source: Table 6 · p. 9, §4.3 · p. 9
• With supervision, MoD's pairwise BLEU score fell substantially, indicating greater specialization among mixture decoders; the comparison used top-five baseline hypotheses and one top hypothesis per MoD mixture.Lower pairwise BLEU indicates greater diversity in the hypothesis set.Source: Table 7 · p. 9, §4.3 · p. 9

• The model combines mixture modeling with sequence-to-sequence generation and introduces a discrete latent variable for interpretable dialog actions or topics.Source: §5 · p. 9
• The authors attribute MoD's superior diversity to decoder specialization and report reasonable automatic-metric performance against other latent-variable and mixture models.Source: §5 · p. 9
• Human A/B evaluation supports the diversity claim, with MoD responses often preferred over baseline responses.Source: §5 · pp. 9–10
• The authors identify more fine-grained linguistic features for supervising MoD as a possible route to further performance improvement.Source: §5 · p. 10