Source-linked AI summary
Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, Zhi Jin
TL;DR
Neural dialogue systems often generate safe, low-content replies, motivating a method that explicitly introduces reply content. The paper predicts a noun keyword with PMI, then uses seq2BF to generate a fluent reply containing it. The approach consistently outperforms pure seq2seq generation in human evaluation and entropy measures, while allowing the keyword to appear at an arbitrary reply position.
Problem
Neural dialogue systems tend to generate safe, universally relevant replies with little meaning, and queries may not provide enough information for informative decoding.
Method
The approach predicts a noun keyword using pointwise mutual information, then uses seq2BF to generate backward and forward sequences conditioned on the keyword and query.
Results
The approach consistently outperforms pure seq2seq models in human evaluation and entropy measures, while allowing the predicted keyword to appear at an arbitrary position.
Takeaways & Limitations
Content-introducing seq2BF produces more meaningful replies than pure seq2seq generation while retaining fluent generation around a selected keyword.
Takeaways & Limitations
The authors identify keyword prediction as an area for improvement and suggest applying different prediction techniques in future work.
Abstract
from arXiv · showhide
Using neural networks to generate replies in human-computer dialogue systems is attracting increasing attention over the past few years. However, the performance is not satisfactory: the neural network tends to generate safe, universally relevant replies which carry little meaning. In this paper, we propose a content-introducing approach to neural network-based generative dialogue systems. We first use pointwise mutual information (PMI) to predict a noun as a keyword, reflecting the main gist of the reply. We then propose seq2BF, a "sequence to backward and forward sequences" model, which generates a reply containing the given keyword. Experimental results show that our approach significantly outperforms traditional sequence-to-sequence models in terms of human evaluation and the entropy measure, and that the predicted keyword can appear at an appropriate position in the reply.
1 Introduction
Open-domain dialogue systems moved from retrieval and statistical approaches toward neural seq2seq generation, but neural models often produce safe replies with little meaning. The paper addresses this problem by introducing reply content through a predicted keyword and a modified generation process.
- Open-domain conversation is difficult for rule-based systems because diverse, uncertain interactions make manually designed rules or templates impractical.
- Neural dialogue studies commonly encode a user query into a vector and decode that representation into a reply with seq2seq models.The encoders and decoders use recurrent neural networks to model word sequences.
- Neural dialogue performance remains unsatisfactory because models favor safe, universally relevant replies such as “something” and “I don’t know.”These replies occur frequently in training data and therefore can receive high estimated probability during decoding.
- The paper predicts a noun keyword with high pointwise mutual information and uses seq2BF to generate a reply containing that keyword.The keyword is intended to reflect the reply’s main gist and introduce substantial content.
2 Our Approach
The approach introduces reply content by predicting a query-relevant noun with PMI, then generating backward and forward sequences conditioned on that keyword and the query. Its design accommodates keywords at different reply positions and trains without additional labels.
- Overview: The system has two steps: PMI predicts a reply keyword, then generation conditions on both the keyword and query.The keyword predictor uses surface statistics because it outputs one word, while the generator introduces content suited to the query.
- Keyword Predictor: PMI favors words mutually informative with the query over common words, and candidate keywords are manually restricted to nouns.The keyword is selected by the highest PMI score against the query words, using an additive word-level approximation based on independence assumptions.
- seq2BF Model: Unlike standard left-to-right decomposition, seq2BF starts from the predicted keyword because it may occur at the beginning, middle, or end of the reply.The keyword splits the reply into backward and forward subsequences, with both generators allowing inner- and inter-subsequence dependencies.
- seq2BF Model: The seq2BF model generates the reply’s backward half from the keyword and query, then generates the forward half conditioned on the same context.The backward decoder outputs the preceding words in reversed order, producing a fluent reversed half relative to the split keyword from a mathematical perspective.
- Training: The model requires no additional labels: training samples a reply word as the split word, reverses its preceding half, and trains separate backward and forward generators.The backward generator must be trained on half replies beginning at the split word; full reversed sentences would cause it to generate an entire reversed reply.
3 Experiments
The experiments evaluate the approach on a Chinese human-conversation dataset using human and intrinsic metrics, alongside seq2seq and keyword-free seq2BF baselines. Content-introducing seq2BF receives the strongest human evaluations, increases entropy, and produces replies with concrete keywords positioned flexibly.
- Experimental Setup: The evaluation uses 500,000 training pairs, 2,000 validation samples, 27,871 test samples, and PMI statistics from 100M query-reply pairs.The data comes from the Baidu Tieba forum, with character-level seq2BF training and noun-phrase keyword prediction.
- Human Evaluation: Human evaluation randomly samples 200 test queries and replies for pointwise and pairwise judgments by six blind volunteers.Pointwise scores assess reply appropriateness, while pairwise judgments compare pure seq2seq with content-introducing seq2BF.
- Human Evaluation: Content-introducing seq2BF achieves the highest pointwise human score and substantially outperforms pure seq2seq in pairwise evaluation.The keyword-free seq2BF performs poorly, indicating that splitting sequences alone does not explain the improvement; annotators agree on the system ranking.
- Intrinsic Metrics: 30% entropy increase: seq2BF with a predicted keyword generates slightly shorter but more information-rich replies than seq2seq.Entropy is an intrinsic metric measuring information in generated utterances; the objective metrics require no reference reply.
- Case Studies and Discussion: Case studies show seq2seq producing generic replies such as “me too,” whereas keyword-conditioned seq2BF generates concrete terms and places the keyword at arbitrary reply positions.The backward-and-forward generation expands around the predicted keyword to form a more complete utterance.
- Intrinsic Metrics: With a noun keyword, remaining words reach average character-level entropy 7.422, versus 6.971 without keywords.Because the seq2BF model is otherwise unchanged, the analysis attributes the difference to content introducing within the reported discussion.
4 Related Work
Related work spans rule- and template-based systems, data-driven open-domain methods, and neural approaches for dialogue generation. Prior work also targets universal replies using mutual information, variational inference, topic information, or external knowledge.
- Rule- and template-based dialogue systems primarily serve vertical domains, while open-domain systems use data-driven approaches because of conversational diversity and uncertainty.
- Seq2seq models encode a source sentence with an RNN and decode it into a target sentence, supporting neural natural-language generation tasks.
- Neural dialogue research includes bag-of-words-conditioned RNNs, attention-based short-text systems, and hierarchical networks for multi-turn conversation.
- To address universal replies, prior studies use mutual-information training, variational Bayes, topic information, or selected knowledge-base triples.
5 Conclusion and Future Work
The paper introduces a content-introducing approach that predicts a noun keyword with PMI and generates a fluent reply containing it through seq2BF. The approach consistently outperforms pure seq2seq on human evaluation and entropy, while future work targets improved keyword prediction and broader applications.
- The method predicts a noun keyword with pointwise mutual information, then uses seq2BF to generate a reply containing that keyword.
- Seq2BF allows the predicted keyword to appear at an arbitrary reply position while maintaining fluency.
- Future work includes alternative keyword prediction techniques and extending seq2BF to generative question answering with external databases or knowledge bases.