Source-linked AI summary
Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, Ray Kurzweil
TL;DR
Long conversation responses remain difficult to generate coherently, informatively, and diversely, especially when length-promoting decoding creates incoherent outputs. The paper combines target-side modeling through a scalable glimpse model with stochastic segment-level reranking, finding longer and better-rated responses than heuristic baselines. Its conclusions are bounded by the memory cost of direct target-side attention and the difficulty of using marginal scoring directly during beam search.
Problem
Long, informative, coherent, and diverse conversation responses remain difficult because standard seq2seq decoding can produce incoherent, redundant, or contradictory long outputs.
Method
The paper combines a scalable fixed-length glimpse model with stochastic beam search and segment-by-segment reranking to model prior target content and inject diversity earlier.
Results
The proposed techniques produce longer responses and higher human ratings than baseline seq2seq models using explicit length and diversity-promoting heuristics.
Takeaways & Limitations
Minimal modeling changes, scalable fixed-length decoding, and slightly more advanced decoding can improve the quality, diversity, and coherence of neural conversation responses.
Takeaways & Limitations
Direct target-side attention is too memory-intensive for large-scale training, and marginal scoring produces usually ungrammatical sequences when used directly during beam-search generation.
Abstract
from arXiv · showhide
Sequence-to-sequence models have been applied to the conversation response generation problem where the source sequence is the conversation history and the target sequence is the response. Unlike translation, conversation responding is inherently creative. The generation of long, informative, coherent, and diverse responses remains a hard task. In this work, we focus on the single turn setting. We add self-attention to the decoder to maintain coherence in longer responses, and we propose a practical approach, called the glimpse-model, for scaling to large datasets. We introduce a stochastic beam-search algorithm with segment-by-segment reranking which lets us inject diversity earlier in the generation process. We trained on a combined data set of over 2.3B conversation messages mined from the web. In human evaluation studies, our method produces longer responses overall, with a higher proportion rated as acceptable and excellent as length increases, compared to baseline sequence-to-sequence models with explicit length-promotion. A back-off strategy produces better responses overall, in the full spectrum of lengths.
1 Introduction
General-purpose conversation remains difficult because responses must be long, informative, coherent, and diverse. This work introduces two minimally invasive methods that improve response length and human ratings over heuristic seq2seq baselines.
- General-purpose conversation systems are challenging but important for agents that interact with humans through natural language.
- Conversation response generation is inherently creative, unlike translation, because many responses can be appropriate for one source.
- The paper proposes a glimpse model for scalable training and segment-based stochastic decoding for earlier diversity injection.
- The two methods produce longer responses and higher ratings than a baseline seq2seq model using explicit length and diversity-promoting heuristics.
2 Overview and Motivation
The paper addresses long-response coherence and diversity by combining target-side context during training with stochastic, segment-level decoding. The resulting methods are designed to scale while improving response quality and supporting future evaluation.
- Motivation: Standard seq2seq models struggle with long responses because a fixed-length decoder state must retain all previously generated content.
- Motivation: Target-side attention can improve coherence by tracking generated content, but its memory cost makes direct scaling to large data sets impractical.
- Glimpse model: The glimpse model trains on fixed-length target segments while placing preceding target symbols on the encoder, providing a scalable trade-off.
- Decoding: Length-promoting decoding increases output length but often sacrifices coherence, while standard MAP beam search tends to produce short, generic responses.
- Decoding: Stochastic beam search with segment-by-segment reranking injects diversity earlier while retaining more coherence than step-by-step sampling.
- Contributions: The paper releases context-free conversation prompts as a benchmark for future open-domain response research.
3 Seq2Seq Model with Attention on Target
The model represents conversation responses as conditional generation from a source sequence, then augments standard attention with previously generated target content. Because full target-side attention is memory-intensive, the paper develops fixed-length target glimpses for scaling.
- Seq2seq formulation: A seq2seq model encodes source sequence x and models the conditional language model P(y|x) for target sequence y.
- Standard attention: Standard attention summarizes source symbols most relevant to predicting the next target symbol from the decoder’s previous state.
- Target-side attention: Target-side attention augments this mechanism with previously generated target symbols through Attention(h_i−1, y[0:i−2], x).
- Target-side attention: On OpenSubtitles 2009, target-side attention reduced perplexity from 24.6 without attention to 24.2 with attention.
- Scaling limitation: Direct target-side attention caused frequent out-of-memory issues in the 8-layer GNMT model when scaling to Reddit.
- Glimpse model: The target-glimpse model uses a fixed-length decoder and trains on contiguous target segments, with preceding target symbols supplied through the encoder.
4 Stochastic Decoding with Segment-by-Segment Reranking
The method generates response segments stochastically and reranks them with normalized scores, continuing segment by segment until termination. Random prompts support the reranking, while small beam width preserves behavior distinct from standard beam search.
- Decoding procedure: Segment-by-segment decoding generates candidate segments with a trained glimpse-model and stochastic beam search, then reranks and selects one segment at each stage.The selected segment is concatenated with the source and used to generate the next candidates until an end-of-sequence token appears.
- Stochastic beam search: Stochastic beam search replaces deterministic top-B selection with sampling to encourage variation and prevent one beam from dominating response diversity.For each beam extension, the algorithm samples D candidate tokens rather than enumerating all possible single-token extensions.
- Segment reranking: The normalized score reranks candidate segments using randomly sampled source sequences, called prompts.In experiments, Q prompts are randomly selected from the context-free evaluation set.
- Score interpretation: When prompts are unbiased samples from P(x), the denominator approximates P(y_k|y_1:k−1) by Monte Carlo sampling.For whole target-sequence reranking, this becomes the marginal P(y), corresponding to a diversity-promoting objective.
- Implementation choices: A small beam width is used because larger values make the stochastic algorithm behave more like standard beam search.The experiments set the number of beams B to 2, with D set to 10 samples per beam.
5 Experimental Results
The experiments evaluate large-scale training, automatic diagnostics, and human judgments of baseline and proposed conversation-generation methods. The proposed approach produces longer responses while preserving or improving ratings as length increases.
- Data: The training corpus combines Reddit, Open Subtitles, Stack Exchange, and web-extracted dialogue, with message pairs treated as single-turn exchanges.Reddit contributes 1.7 billion messages across 221 million conversations; the other sources contribute 0.5 million, 0.8 million, and 17 million conversations, respectively.
- Evaluation: Human evaluation rates each prompt-response pair on a five-level scale from Bad to Excellent, using three raters per pair across 200 pairs per method.The evaluation prompts are context-free, open-domain, and drawn from internal testing, Fisher, and Jabberwacky sources.
- Baseline: Length normalization makes baseline responses longer but more semantically incoherent, with longer outputs receiving more Bad and fewer Excellent or Good ratings.The baseline uses α = 0.8 length normalization; the reported example is “I have no idea what you are talking about.”
- Proposed method: One third of proposed responses exceed 100 characters, while Acceptable and Excellent ratings remain constant or increase with length; baseline ratings decline sharply.The proposed method uses a length-10 target-glimpse model with stochastic beam search and segment-by-segment reranking.
- Agreement: 80% of baseline responses and 70% of proposed-method responses achieve major agreement among raters.Major agreement means that two of the three ratings are the same.
- Overall comparison: Among 180 preferred response pairs, raters chose the combined model 103 times versus 77 for the baseline, while 20 pairs had no preference.The combined strategy produced more Excellent, Good, Acceptable, and Mediocre ratings and fewer Bad ratings.
6 Conclusion
The paper concludes that modest modeling and decoding changes, combined with very large datasets, noticeably improve neural conversation responses. Fixed-length decoder training supports earlier improvements in diversity and coherence, with possible relevance beyond conversation modeling.
- Conclusion: Minimal modeling changes, advanced decoding, and very large datasets can produce noticeable improvements in neural conversation response quality.The authors describe general-purpose conversation research as still being in its infancy.
- Conclusion: Fixed-length decoder training facilitates large-scale training and improves diversity and coherence earlier during generation.The authors expect some results to carry over to machine translation and image-captioning.
A The Context-Free Prompt List and Evaluation Results
The context-free prompt evaluation presents randomly selected responses from the proposed method and baseline across five human-rating levels. The examples illustrate differences in relevance, coherence, repetition, and response length across the two methods.
- A.1 Our Method: The proposed-method examples include responses ranging from brief relevant answers to longer attempts at conversational engagement.Examples include “I want to be a psychologist,” a supportive response to distress, and longer responses about food, weather, and daily life.
- A.1 Our Method: Some proposed-method responses remain incoherent or contradictory despite being longer.The steamer response repeats conflicting ownership claims, while other examples mismatch prompts or contain nonsensical content.
- A.2 Baseline Method: The baseline also produces occasional relevant answers, including responses about novels, dentists, Mars, work, and location.These examples show that baseline outputs are not uniformly poor, although many remain brief.
- A.2 Baseline Method: Figure 2 reports response counts above character-length thresholds and the proportions rated at least Acceptable or Excellent.The figure also includes five-rating histograms and training-progress curves for Length-1, Length-10, Plain Seq2seq, and Length-10 with an encoder-to-decoder connection.
- A.2 Baseline Method: The displayed methods include beam search on plain seq2seq, length-normalized beam search, and segment beam sampling with the glimpse model.The figure distinguishes the proposed glimpse-model variants from the Plain Seq2seq baseline.