Source-linked AI summary
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, Steve Young
TL;DR
NLG systems often rely on rules that produce rigid responses and do not scale easily across domains and languages. This paper introduces a semantically controlled LSTM that jointly learns sentence planning and surface realisation from unaligned data. Across two domains, it improved objective performance, while human judges rated it more informative and natural and preferred it overall.
Problem
Rule- and heuristic-based NLG often produces rigid, repetitive responses and does not easily scale across domains and languages.
Method
A semantically controlled LSTM jointly optimises sentence planning and surface realisation from unaligned data using cross entropy training.
Results
The SC-LSTM achieved the best overall performance on two objective metrics across two domains and was strongly preferred by human judges.
Takeaways & Limitations
The unified corpus-based architecture supports natural linguistic variation and may reduce the cost and lifecycle of domain-scalable and multilingual NLG.
Takeaways & Limitations
The ERR reranking criterion cannot handle arbitrary slot-value pairs, including binary slots or slots with a don’t care value.
Abstract
from arXiv · showhide
Natural language generation (NLG) is a critical component of spoken dialogue and it has a significant impact both on usability and perceived quality. Most NLG systems in common use employ rules and heuristics and tend to generate rigid and stylised responses without the natural variation of human language. They are also not easily scaled to systems covering multiple domains and languages. This paper presents a statistical language generator based on a semantically controlled Long Short-term Memory (LSTM) structure. The LSTM generator can learn from unaligned data by jointly optimising sentence planning and surface realisation using a simple cross entropy training criterion, and language variation can be easily achieved by sampling from output candidates. With fewer heuristics, an objective evaluation in two differing test domains showed the proposed method improved performance compared to previous methods. Human judges scored the LSTM system higher on informativeness and naturalness and overall preferred it to the other systems.
1 Introduction
The paper targets scalable, less rigid NLG by replacing handcrafted generation decisions with a semantically controlled LSTM that learns planning and realisation from data.
- Rule-based generators are robust and adequate but often repeat identical, stilted forms and scale poorly to large open-domain systems.
- Trainable modular systems adapt across domains or styles but still require handcrafted generators to define their statistical decision space.
- Corpus-based methods learn generation directly from data, using flexible structures and candidate reranking to produce more natural responses.
- The proposed SC-LSTM jointly optimises sentence planning and surface realisation from unaligned data with cross entropy and samples outputs for variation.
- Evaluation covers restaurant and hotel domains, testing objective performance and subjective quality against several existing generation methods.
2 Related Work
Prior NLG research separated planning from realisation and progressively reduced handcrafted rules through corpus-based and neural methods, but annotation and control burdens remained.
- Conventional NLG separates sentence planning, which maps semantics to an intermediate form, from surface realisation, which produces final text.
- Some statistical planners use corpus-derived derivations or reinforcement learning, but mapping plans to valid syntax can still require handcrafting.
- Corpus-based NLG reduces dependence on rules through language models, phrase-based systems, active learning, and template matching.
- Template matching can improve outputs with reranking but does not generalise well to all generation settings.
- Neural NLG exploits distributed representations and long dependencies, while earlier recurrent systems still needed heuristic gates to preserve slot information.
- LSTM addresses difficult long-range training by using memory blocks and multiplicative gates in recurrent connections.
3 The Neural Language Generator
The neural generator combines recurrent token prediction with semantic control so sentence planning and surface realisation jointly preserve dialogue-act information. It extends this design with deep layers, backward reranking, cross-entropy training, and slot-aware decoding.
- Neural language generation: The generator predicts each next token from the current hidden state, sampling tokens sequentially until an utterance is produced.Slot values are replaced by slot tokens during generation and can later be lexicalised.
- Semantic control: SC-LSTM adds a control cell that retains or discards dialogue-act features during generation, linking sentence planning to surface realisation.The control vector represents the dialogue act and its slot-value pairs.
- Semantic control: The reading gate learns token-pattern associations with slots and uses them to affect dialogue-act features inside the network.The gate is described as a detector of keywords and key phrases associated with particular slots.
- Deep structure: Deep variants stack multiple LSTM layers above the dialogue-act cell, using skip connections and dropout to address vanishing gradients and overfitting.Dropout is applied to non-recurrent connections rather than word embeddings.
- Backward reranking: A backward SC-LSTM reranker uses backward context to select among candidates generated by the forward model.The approach avoids directly integrating bidirectional information into the sequential generator.
- Training: Training uses cross-entropy between predicted and actual word distributions, with additional regularisation for dialogue-act and reading-gate dynamics.The slot-completeness penalty discourages outputs that fail to render required slots, while another term discourages multiple gates switching off simultaneously.
- Decoding: Decoding over-generates candidates by random sampling, then reranks them using forward and backward costs together with slot error rate.Slot error rate is computed by exact matching of slot tokens and penalises missing or redundant slots.
4 Experiments
The experiments evaluated SC-LSTM generators across restaurant and hotel domains using objective metrics, human quality ratings, and pairwise preferences. The deep SC-LSTM achieved the strongest overall performance, while SC-LSTM systems were generally preferred by human judges.
- Experimental Setup: The experiments covered restaurant and hotel domains, with 248 distinct dialogue acts in restaurants and 164 in hotels.Each domain used 12 attributes, and the collected corpora contained around 5K system utterances per domain.
- Objective Evaluation: The study compared shallow and deep SC-LSTM generators with handcrafted, kNN, class-based language-model, gated RNN, and ungated RNN/LSTM baselines.The deep variant used two hidden layers, while the shallow model used one.
- Objective Evaluation: The deep SC-LSTM achieved the best overall objective performance by combining learned semantic gates with a deep architecture.The experiments also found that gated models had much lower slot error rates than ungated models.
- Human Evaluation: SC-LSTM systems outperformed the class-based language models and heuristic-gated RNN in both human-rated informativeness and naturalness.The deep SC-LSTM was significantly better than class-based language models for informativeness and than the heuristic-gated RNN for naturalness.
- Human Evaluation: Human judges significantly preferred the SC-LSTM systems, with a strong preference for the deep model over the other systems but not significantly over the shallow SC-LSTM.Human evaluation tested 1,000 dialogue acts and approximately 1,300 generated utterances per system after filtering identical outputs.
5 Conclusion and Future Work
The paper presents SC-LSTM as a unified, end-to-end trainable generator that produces linguistically varied responses from data. It reports strong objective and human-evaluation results, while identifying domain scalability and multilingual NLG as future directions.
- Conclusion: SC-LSTM is a deep, semantically controlled LSTM architecture for generating natural, linguistically varied responses.The proposed generator models NLG in a unified architecture trained end-to-end from data.
- Conclusion: The generator jointly optimises sentence planning and surface realisation on unaligned data using cross entropy without heuristics or handcrafting.This combines the two generation components within one trainable neural architecture.
- Evaluation: SC-LSTM achieved the best overall performance on two objective metrics across two different domains, and human judges strongly preferred it to existing methods.The conclusion reports both objective performance and human preference as evidence for the approach.
- Future Work: Future work will examine conditioning on discourse information or social cues and extending corpus-based NLG toward scalable multilingual systems.The paper identifies domain scalability and multilingual NLG as future applications of the corpus-based approach.