Source-linked AI summary
A Diversity-Promoting Objective Function for Neural Conversation Models
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan
TL;DR
Neural conversation models often produce safe, commonplace responses because likelihood of the response given the message is poorly suited to response generation. The paper replaces this objective with Maximum Mutual Information, and reports more diverse, interesting responses with improved BLEU and human-evaluation quality. The approach is subject to practical decoding difficulties and dataset scope limits.
Problem
Neural conversation models tend to generate safe, commonplace responses regardless of the input, motivating an alternative to unidirectional response-given-message likelihood.
Method
The paper uses Maximum Mutual Information, which measures mutual dependence between inputs and outputs, as the neural response-generation objective.
Results
MMI models produce more diverse and interesting responses while improving quality measured by BLEU and human evaluation.
Takeaways & Limitations
MMI reduces generic responses and yields more varied outputs in conversational response generation.
Takeaways & Limitations
MMI-antiLM can produce ungrammatical responses and MMI-bidi can make decoding intractable, while IMSDB is too small for open-domain dialogue training.
Abstract
from arXiv · showhide
Sequence-to-sequence neural network models for generation of conversational responses tend to generate safe, commonplace responses (e.g., "I don't know") regardless of the input. We suggest that the traditional objective function, i.e., the likelihood of output (response) given input (message) is unsuited to response generation tasks. Instead we propose using Maximum Mutual Information (MMI) as the objective function in neural models. Experimental results demonstrate that the proposed MMI models produce more diverse, interesting, and appropriate responses, yielding substantive gains in BLEU scores on two conversational datasets and in human evaluations.
1 Introduction
Neural conversation models can generate grammatical responses yet often favor generic, non-committal outputs. The paper proposes Maximum Mutual Information (MMI) to reduce this behavior and produce more varied responses.
- Motivation: Neural conversation models often generate high-frequency, generic responses such as “I don’t know” or “I’m OK.”Top-ranked SEQ2SEQ outputs are generic even though more meaningful responses appear lower in the N-best list.
- Motivation: Conversational response systems should produce responses that are grammatical, coherent, diverse, and interesting.
- Proposed approach: MMI measures mutual dependence between inputs and outputs and is proposed as the neural generation objective.The paper presents practical strategies for applying MMI in neural generation models.
- Expected effect: MMI reduces the proportion of generic response sequences while generating more varied and interesting outputs.
2 Related work
Related work has framed response generation as statistical machine translation and increasingly applied neural sequence-to-sequence models. This paper instead targets a single non-trivial response using MMI rather than diversity across multiple outputs.
- Response-generation paradigms: Data-driven, end-to-end response generation contrasts with conventional dialog systems that are typically template- or heuristic-driven.
- Neural response generation: Neural response-generation research progressed from SMT systems with neural reranking toward direct end-to-end SEQ2SEQ models.
- Diversity objectives: Unlike prior methods producing mutually diverse outputs, this work aims to produce a single non-trivial response without identifying lexical overlap.
- Diversity objectives: The paper applies MMI specifically to response generation and compares MMI models against maximum likelihood.
3 Sequence-to-Sequence Models
The SEQ2SEQ model uses LSTMs to encode input sequences and predict output tokens sequentially. Decoding can use greedy search or beam search, with each next token selected from conditional probabilities.
- LSTM architecture: An LSTM represents each input time step using input, memory, and output gates, along with hidden and cell-state vectors.
- Output generation: SEQ2SEQ generation pairs each input sequence with an output sequence and predicts output tokens sequentially using a softmax distribution.
- Output generation: Input and output sequences commonly use separate LSTMs with distinct compositional parameters.
- Decoding: During decoding, greedy search selects the highest-probability token, while beam search maintains multiple candidate sequences.Decoding terminates when an EOS token is predicted.
4.1 Notation
The response-generation notation defines a source message sequence and a target response sequence terminated by an end-of-sentence token, with word embeddings drawn from a vocabulary.
- Notation: S denotes the source message sequence, containing Ns words.
- Notation: T denotes the target response sequence, containing Nt tokens followed by EOS.
- Notation: Each target word token t has a distinct D-dimensional embedding, and V denotes vocabulary size.
4.2 MMI Criterion
The standard sequence-to-sequence objective favors targets likely given the source, which can produce generic responses. The paper replaces it with MMI, balancing source-to-target and target-to-source dependence while penalizing generic responses.
- The standard objective maximizes the log-likelihood of target T given source S.
- MMI instead maximizes mutual information between source S and target T.
- The generalized MMI objective uses λ to control the penalty applied to generic responses.
- Using Bayes’ theorem, the objective trades off targets given sources, p(T|S), against sources given targets, p(S|T).
- Because joint MMI training is empirically nontrivial and repeatedly retraining models is costly, the authors train maximum-likelihood models and apply MMI only during testing.
4.3 Practical Considerations
The two MMI variants require practical adaptations because direct decoding can produce ungrammatical outputs or become intractable. The authors use a length-aware anti-language-model formulation and N-best reranking, which improve interestingness and diversity in practice.
- MMI-antiLM and MMI-bidi are difficult to apply directly because the former can produce ungrammatical responses and the latter makes decoding intractable.
- MMI-antiLM: The anti-language-model term can penalize fluent responses and cause the model to select ungrammatical outputs.
- MMI-antiLM: The anti-language model is modified by replacing p(T) with U(T), which applies a monotonically decreasing weight to later token positions.
- MMI-antiLM: This position-dependent penalty reflects the diminishing influence of the source representation as previously predicted words accumulate during decoding.
- MMI-bidi: Direct MMI-bidi decoding is infeasible because p(S|T) requires completing the target before it can be computed over the enormous target search space.
- MMI-bidi: The practical approximation first generates N-best lists with p(T|S), then reranks them using the second objective term, preserving likely grammaticality but sacrificing global optimality.
- The authors report that both MMI variants significantly improve interestingness and diversity in practice.
4.4 Training
The training setup uses a deep LSTM sequence-to-sequence architecture with four layers for both encoding and decoding. Optimization uses fixed-rate stochastic gradient descent, batching, clipping, and a single-GPU implementation.
- The model uses four LSTM layers for encoding and four for decoding, with 1,000 hidden neurons per layer and 1,000-dimensional word embeddings.
- Parameters and embeddings are initialized uniformly in [−0.08, 0.08].
- Training uses stochastic gradient descent with a fixed learning rate of 0.1 and batch size 256.
- Gradients are clipped by scaling them when their norm exceeds 1.
- The implementation processes approximately 600-1200 tokens per second on a single Tesla K40 GPU.
- The reverse model p(S|T) uses the same architecture as p(T|S), with messages and responses interchanged.
4.5 Decoding
Decoding combines the conditional response score with a length penalty. The system generates and reranks candidate responses, tuning the penalty weights on development data.
- Decoding combines the loss function with length penalization to score a target response T.
- The length term uses target length N_t and weight γ.
- MERT tunes λ and γ on the development set.
- Beam search uses beam size B = 200, preserves the top B unfinished hypotheses, and limits generated candidates to length 20.
- The system generates N-best lists from p(T|S), then reranks them using p(T|S), λp(S|T), and γN_t.
5 Experiments
Experiments evaluate MMI models on Twitter and OpenSubtitles using BLEU, diversity metrics, qualitative examples, and human judgments. MMI improves response diversity and performance, while results depend on dataset properties and the diversity of candidate lists.
- 5.1 Datasets: The experiments use Twitter and OpenSubtitles datasets, with IMSDB serving as the evaluation source where OpenSubtitles lacks reliable speaker-turn information.Twitter contains 23 million snippets, while OpenSubtitles contains roughly 60M–70M scripted lines; IMSDB provides character identities but is too small for open-domain training.
- 5.2 Evaluation: Evaluation combines BLEU with distinct-1 and distinct-2, which measure normalized unigram and bigram diversity in generated responses.Twitter uses multi-reference BLEU, whereas OpenSubtitles uses single-reference BLEU because IMSDB is too limited for multiple references.
- 5.3 Results: On Twitter, MT+neural reranking reaches BLEU 4.44, while MMI-bidi provides a significant BLEU and diversity improvement over baseline SEQ2SEQ.The MT-based systems were trained on roughly 50 million examples, substantially more than the neural-model experiments described here.
- 5.3 Results: MMI-antiLM raises OpenSubtitles BLEU by up to 36% and increases unigram diversity by more than 200% over the baseline.The reported improvement is associated with a significant performance boost for MMI-antiLM on the OpenSubtitles dataset.
- 5.3 Results: MMI-bidi outperforms baseline, greedy SEQ2SEQ, SMT, and SMT+RNN systems in pairwise human evaluations at α = 0.05.Its isolated overall-quality rating is 3.84 on a 5-point scale, based on evaluations by seven judges per output.
- 5.3 Results: MMI produces more diverse and interesting responses than SEQ2SEQ, whereas standard models become conservative and duller as message complexity increases.Four generic strings account for 32% of all generated responses in one reported analysis.
6 Conclusions
The paper addresses generic conversational responses by replacing the unidirectional response-given-message objective with Maximum Mutual Information. Its MMI models produce more diverse and interesting responses while improving BLEU and human-evaluation quality, but grounding, persona, and intent remain future work.
- The analysis attributes generic responses at least partly to optimizing the unidirectional likelihood of responses given messages.
- MMI replaces this objective by optimizing mutual information between conversational inputs and outputs.
- MMI models produce more diverse and interesting responses while improving quality on BLEU and human evaluation.
- The study focuses on algorithmic factors, leaving grounding, persona, and user or agent intent for future investigation.
- Human annotators evaluated the overall quality of responses in their particular conversations.
- The paper suggests implications beyond dialogue for tasks requiring mutual correspondences, including image description and question answering.