Source-linked AI summary
Natural Language Generation in Dialogue using Lexicalized and Delexicalized Data
Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, Philip Bachman
TL;DR
Task-oriented dialogue generation needs to turn dialogue-act slot-value pairs into well-formed sentences, but prior RNN approaches may ignore lexical values or fail to align them with slots. This paper encodes both lexicalized values and delexicalized slots, and reports stronger results across two datasets, with further gains from pretrained sentence auto-encoder weights.
Problem
RNN-based dialogue generation can ignore lexical values or leave them unaligned with slots, missing linguistic relationships that matter when producing well-formed sentences.
Source: §1 · p. 1Method
Results
The models consistently outperformed existing RNN-based approaches on the CF restaurant and DSTC2 datasets, and transfer learning can yield better responses.
Source: §4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Natural language generation plays a critical role in spoken dialogue systems. We present a new approach to natural language generation for task-oriented dialogue using recurrent neural networks in an encoder-decoder framework. In contrast to previous work, our model uses both lexicalized and delexicalized components i.e. slot-value pairs for dialogue acts, with slots and corresponding values aligned together. This allows our model to learn from all available data including the slot-value pairing, rather than being restricted to delexicalized slots. We show that this helps our model generate more natural sentences with better grammar. We further improve our model's performance by transferring weights learnt from a pretrained sentence auto-encoder. Human evaluation of our best-performing model indicates that it generates sentences which users find more appealing.
1 INTRODUCTION
Task-oriented dialogue generation has moved beyond hand-crafted templates, but recurrent models can still produce ungrammatical output when they ignore lexicalized slot values or fail to align values with slots. The paper proposes using both lexicalized and delexicalized slot-value pairs, and reports further improvement from transferring pretrained sentence auto-encoder weights.
1 INTRODUCTION
- Existing RNN approaches either ignore lexical values or fail to align values with slots, limiting their access to linguistic relationships needed for grammatical output.For example, the lexical value “near the plaza” contains relations that a delexicalized slot label alone does not capture.Source: §1 · p. 1
- The paper proposes an RNN-based approach that uses both lexicalized and delexicalized dialogue-act slot-value pairs.The method is motivated by template-based systems that do not scale well to complex domains and datasets.Source: §1 · p. 1
- Transferring weights from a pretrained sentence auto-encoder further improves the model’s performance.Source: §1 · p. 2
2 MODEL
The model encodes aligned lexicalized and delexicalized slot-value pairs, then decodes them into a sentence. Its training objective encourages accurate word prediction and complete, orderly realization of the required dialogue acts.
2 MODEL
- The bidirectional LSTM encoder represents each slot-value pair by concatenating a one-hot delexicalized-slot vector with the mean embedding of its lexicalized value.Source: §2 · p. 2
- The decoder initializes its hidden state and memory from an average-pooled encoder context, then predicts words or delexicalized slots until EOS or a maximum length.Generated slots are replaced with their lexicalized values; test-time decoding uses beam search with width 10 and a slot-error penalty parameter of 1000.Source: §2 · pp. 2–3
- The negative log-likelihood objective adds regularization that encourages required slots to be emitted and discourages dropping multiple dialogue-act elements at once.The regularization scalars are η = 0.0001 and ξ = 100.Source: §2 · p. 3
- For limited-data domains, the transfer model initializes its decoder’s hidden weights from a sentence auto-encoder trained to reconstruct same-topic sentences, then fine-tunes them.Source: §2 · p. 3
3 RESULTS
The proposed models outperform the baselines on automated metrics and human evaluations, while qualitative examples show more grammatical responses from models that use lexicalized slot values.
3 RESULTS
- Both ld-sc-LSTM and transfer-ld-sc-LSTM consistently outperform the other baselines on automated metrics.The reported metrics are BLEU-4, METEOR, ROUGE L, and CIDEr.Source: §3 · p. 3
- Human judges' ratings statistically favor the authors’ models over baselines on both datasets.Transfer-ld-sc-LSTM scores higher than ld-sc-LSTM on DSTC2, but that difference is not statistically significant.Source: §3 · p. 3
- Automated metrics do not strongly correlate with human scores, so the two evaluation approaches need not agree.The passage attributes this observation to Liu et al. (2016).Source: §3 · p. 4
- For a “pizza” slot value, lexicalized models avoid adding “food” after “pizza,” unlike the LSTM and d-sc-LSTM outputs.Table 2 compares responses generated for the same dialogue acts.Source: §3 · p. 4, Table 2 · p. 4
- The models are less prone to grammatical errors, including around generated slots, and this is confirmed by human assessment scores.One example contrasts “No Japanese under 30 dollars” with more grammatical responses from the proposed models.Source: §3 · p. 4
4 CONCLUSION
The paper concludes that learning from aligned lexicalized and delexicalized tokens yields a more powerful recurrent encoder-decoder NLG model. It reports consistent gains on two datasets and better responses from transfer learning with a pretrained sentence auto-encoder.
4 CONCLUSION
- The recurrent encoder-decoder model learns from both lexicalized and delexicalized tokens, with slots aligned to values.The conclusion says exposing the network to this complete data led to a more powerful model.Source: §4 · p. 4
- The models consistently outperform existing RNN-based approaches on the CF restaurant and publicly available DSTC2 datasets.Source: §4 · p. 4
- Bootstrapping with weights from a pretrained sentence auto-encoder can result in better generated responses.Source: §4 · p. 4
A DATASETS AND HUMAN EVALUATION
The appendix describes the CF and DSTC2 datasets, their dialogue-act and slot-value settings, and the human evaluation procedure.
A DATASETS AND HUMAN EVALUATION
- CF is a CrowdFlower restaurant-search dataset collected through separate tasks for six dialogue acts, with seven listed slot types.Results use a stratified 85%/15% train/test split, and human experts tagged the dialogue-act slot-value pairs.Source: A DATASETS AND HUMAN EVALUATION · p. 6
- DSTC2 contains templated machine responses annotated with dialogue acts and slot-value pairs, using the CF slot types.Its listed dialogue acts include inform, offer, request, confirmations, canthelp, select, welcome message, repeat, and reqmore.Source: A DATASETS AND HUMAN EVALUATION · p. 6
- In both datasets, request acts allow only empty-valued slots, while other acts also allow dontcare values; 10% of training data is used for validation.Table 3 provides statistics for CF and DSTC2.Source: A DATASETS AND HUMAN EVALUATION · p. 6, Table 3 · p. 6
- Human evaluation sampled 100 test-set dialogue acts per dataset and had five judges rate each model’s top response on a 1-to-5 appropriateness scale.Four model responses were presented with the corresponding dialogue acts in each trial, in randomized order.Source: A DATASETS AND HUMAN EVALUATION · p. 6
B TRAINING DETAILS
Training settings were selected or set as described, with dropout applied at non-recurrent connections.
B TRAINING DETAILS
- Validation data set the decoder layer count and hidden-state dimension, encoder hidden-state dimension, and word-embedding dimensionality.Source: B TRAINING DETAILS · p. 6
- The sc-LSTM reading coefficient α is 1, and the maximum sequence length T is 30.Source: B TRAINING DETAILS · p. 6
- Training uses Adam and dropout of 0.5 at all non-recurrent connections.Source: B TRAINING DETAILS · p. 6