Source-linked AI summary
Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking
Tsung-Hsien Wen, Milica Gasic, Dongho Kim, Nikola Mrksic, Pei-Hao Su, David Vandyke, Steve Young
TL;DR
Spoken-dialogue NLG is costly because it depends on handcrafting or scarce semantic annotations. The paper proposes a jointly optimized recurrent and convolutional generator trained without semantic alignments, and reports efficient, high-quality, natural, and varied outputs compared with n-gram and rule-based systems.
Problem
Spoken-dialogue NLG remains dependent on costly handcrafting or difficult-to-collect semantic annotations, while generated utterances must accurately convey dialogue-act concepts.
Method
The model jointly optimizes a forward RNN generator, CNN reranker, and backward RNN reranker using dialogue act–utterance pairs without semantic alignment or heavy handcrafting.
Results
The system produced high-quality, natural, and linguistically varied utterances, while objective evaluation reported greater efficiency than the compared n-gram generator.
Takeaways & Limitations
Human evaluation indicates that the proposed generator can produce varied utterances preferred to rule-based outputs, while avoiding errors in rendering all dialogue-act information reported for the n-gram generator.
Takeaways & Limitations
The delexicalization approach does not work for some slot-value pairs, and the RNN generator relies only on preceding history for forms needing backward context.
Abstract
from arXiv · showhide
The natural language generation (NLG) component of a spoken dialogue system (SDS) usually needs a substantial amount of handcrafting or a well-labeled dataset to be trained on. These limitations add significantly to development costs and make cross-domain, multi-lingual dialogue systems intractable. Moreover, human languages are context-aware. The most natural response should be directly learned from data rather than depending on predefined syntaxes or rules. This paper presents a statistical language generator based on a joint recurrent and convolutional neural network structure which can be trained on dialogue act-utterance pairs without any semantic alignments or predefined grammar trees. Objective metrics suggest that this new model outperforms previous methods under the same experimental conditions. Results of an evaluation by human judges indicate that it produces not only high quality but linguistically varied utterances which are preferred compared to n-gram and rule-based systems.
1 Introduction
Statistical NLG for spoken dialogue systems remains limited by difficult-to-collect semantic annotations, while conventional systems rely on costly handcrafting. This paper introduces a neural generator trained directly on dialogue act–utterance pairs and evaluates it against established generators.
- Handcrafting makes conventional spoken dialogue systems expensive to build.
- Data-driven NLG remains relatively unexplored because semantically annotated corpora are difficult to collect, leaving rule-based generation as the norm.
- The proposed neural NLG system can be trained on dialogue act–utterance pairs without semantic alignments.
- The generator uses an RNN with dialogue-act conditioning and a controlling gate to over-generate candidate utterances for reranking.
- A CNN reranker checks candidate semantic consistency, while a backward RNNLM reranker improves output fluency.
- In restaurant-information dialogues, objective metrics and human evaluations found the approach more efficient, more natural than a rule-based generator, and linguistically varied.
2 Related Work
Earlier NLG systems commonly decomposed generation into sentence planning and surface realization, while statistical methods reduced handcrafting through language models, structured generation, or reranking. Neural approaches were relatively unexplored, and dialogue NLG additionally required accurate realization of dialogue-act concepts.
- Conventional NLG divides generation into sentence planning and surface realization.
- Statistical NLG methods reduced handcrafting through n-gram generation and later used syntactic dependency trees to address over-generation limitations.
- Other statistical approaches learned sentence-planning decisions with corpus derivations, reinforcement learning, log-linear models, or template extraction and reranking.
- Neural network-based approaches to NLG were relatively unexplored before this work.
- Dialogue NLG requires generated utterances to convey the concepts encoded in the abstract dialogue act accurately.
- CNN-based sentence models were investigated because unconstrained word-level RNNLMs were described as poor at preserving dialogue-act concepts.
3 Recurrent Generation Model
The model generates delexicalised utterances from dialogue-act features, then reranks candidates for semantic consistency and fluency using CNN and backward-RNN models.
- Recurrent generation: Delexicalisation replaces slot values with corresponding slot symbols before training and lexicalises them back into slot values after generation.This allows training on utterance text while preserving slot information for post-processing.
- Recurrent generation: The RNN generator predicts each next token while conditioning on the preceding sequence and a dialogue-act control vector.The control vector concatenates one-hot encodings of the dialogue-act type and associated slot-value pairs and is reapplied at every time step.
- Recurrent generation: A gated control vector decays each slot segment after its first appearance, reducing duplicate slot symbols and encouraging complete rendering of the intended information.The decay factor δ is at most 1, and the gate is controlled using the time each slot first appears.
- Convolutional reranking: The CNN reranker classifies each candidate's dialogue-act type and whether required slots are mentioned, addressing cases that cannot be directly delexicalised.It builds an utterance matrix from token embeddings, applies convolutional feature detectors, averages their outputs over time, and feeds the pooled vectors to a fully connected classifier.
- Reranking: A backward RNNLM reranks candidates using reverse-order context because some sentence forms depend on information beyond the preceding history.The complete architecture therefore trains a forward RNN generator, a CNN reranker, and a backward RNN reranker.
- Decoding: Decoding first over-generates candidates by sampling from the forward RNN and then combines forward likelihood, CNN slot-loss, and backward-RNN likelihood for reranking.The final criterion can additionally include ERR, the number of redundant or missing slots, with λ set to 100 to penalise nonsensical utterances.
4 Experiments
The experiments evaluate the RNN-based generator for San Francisco restaurant dialogue against handcrafted, kNN, and n-gram baselines, using automated metrics, human judgments, and component analyses. The results show strong slot accuracy, BLEU performance, human preference, and benefits from feature gating, CNN reranking, backward reranking, and sufficient training data.
- 4.1 Experimental Setup: The system targets restaurant-information dialogue in San Francisco, covering 8 dialogue-act types and 12 attributes.The attributes are categorical except for the binary kidsallowed slot.
- 4.1 Experimental Setup: The corpus contains 5193 hand-crafted system utterances collected from dialogue turns and is partitioned into training, validation, and test sets in a 3:1:1 ratio.The experiments also use up-sampling because action-type and slot-value frequencies differ markedly across the corpus.
- 4.2 Empirical Comparison: The RNN generator significantly outperforms the handcrafted and kNN approaches, while the O&R system reaches BLEU 0.76 only after partitioning the corpus into increasingly many utterance classes.The RNN produces utterances without slot errors under the same candidate count and achieves the highest BLEU score.
- 4.2 Empirical Comparison: The RNN remains better than O&R on both metrics as the selection beam increases, with slot error reaching zero at a beam of around 50.Sampling from top-5 ranked results is used to introduce linguistic diversity, and the result suggests that over-generation can produce paraphrases.
- 4.3 Human Evaluation: Human judges preferred RNN1 and RNN5 to the rule-based generator, while RNN5 was significantly more informative than O&R.RNN systems scored higher on informativeness and naturalness; RNN5 was preferred overall to RNN1 because of its greater output diversity.
- 4.4 Analysis: Feature gating substantially improves BLEU and slot error, CNN reranking improves performance on hard-to-delexicalize slots, and backward RNN reranking gives stable BLEU gains.The CNN provides a 1.5% improvement on the hard dataset compared with 1% on all data; about 50% of the data suffices for 1-best output, whereas top-5 diversity requires more than 4156 utterances.
5 Conclusion and Future Work
The paper presents a jointly optimized neural NLG model that generates dialogue-act-conditioned utterances without semantic alignment or extensive handcrafting. Human evaluation found the outputs high quality and linguistically varied, while future work targets adaptation to new domains with less data.
- The model jointly optimizes forward RNN generation, CNN reranking, and backward RNN reranking conditioned on the required dialogue act.
- It can be trained on dialogue act-utterance pairs without semantic alignment or heavy feature engineering and handcrafting.
- Human judges found the generated utterances both high quality and linguistically varied.
- The proposed generator is intended for a broader open-domain spoken-dialogue program, with future work focused on transfer to new domains using less adaptation data.The paper suggests that distributed representations and compact parameterization may support this transfer.