Source-linked AI summary
Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings
Ondřej Dušek, Filip Jurčíček
TL;DR
Spoken-dialogue NLG must convert dialogue-manager meaning representations into fluent sentences, with sentence planning and surface realization either separated or jointly modeled. This paper develops a sequence-to-sequence generator that produces strings or deep syntax trees and directly compares both setups. The joint setup performs better on n-gram-based scores while maintaining similar semantic-error performance and producing more relevant outputs.
Problem
Spoken-dialogue NLG must convert meaning representations into sentences, but systems differ between separate sentence planning and surface realization and joint generation.
Method
A sequence-to-sequence generator produces either natural-language strings directly or deep syntax trees later processed by an external surface realizer.
Results
The joint setup achieves significantly higher n-gram-based scores with a similar number of semantic errors and surpasses prior results while reducing irrelevant output information.
Takeaways & Limitations
Both generation modes can be trained successfully with little data, but direct string generation is the more favorable setup in the reported comparison.
Takeaways & Limitations
The comparison with Mairesse et al. (2010) is not directly comparable because that work uses manual alignments.
Abstract
from arXiv · showhide
We present a natural language generator based on the sequence-to-sequence approach that can be trained to produce natural language strings as well as deep syntax dependency trees from input dialogue acts, and we use it to directly compare two-step generation with separate sentence planning and surface realization stages to a joint, one-step approach. We were able to train both setups successfully using very little training data. The joint setup offers better performance, surpassing state-of-the-art with regards to n-gram-based scores while providing more relevant outputs.
1 Introduction
The paper introduces a sequence-to-sequence NLG system that directly compares separate sentence planning and surface realization with joint generation. It produces either natural-language strings or deep syntax trees and achieves strong results with limited training data.
- 1 Introduction: The system supports both two-step generation through deep syntax trees and joint one-step generation directly into natural-language strings.This enables a direct comparison using the same underlying NLG system.
- 1 Introduction: It uses sequence-to-sequence generation with beam search and an n-best reranker to suppress irrelevant output information.The system can be trained from unaligned meaning-representation and sentence pairs alone.
- 1 Introduction: Both strings and deep syntax trees can be learned successfully from much less training data than recent recurrent-neural-network systems use.The experiments use the BAGEL restaurant-information dataset.
- 1 Introduction: The generator surpasses earlier n-gram-based scores while producing more relevant outputs.The reported comparison is against the generator of Duˇsek and Jurˇc´ıˇcek (2015).
2 Generator Setting
Dialogue acts provide the input meaning representation, and the generator can produce either deep syntax trees or natural-language strings. The tree mode separates sentence planning from surface realization, while the string mode performs both jointly.
- 2 Generator Setting: Dialogue acts encode an action such as inform or request together with slot attributes and their values.They serve as the generator’s input meaning representation.
- 2 Generator Setting: The tree-generation mode produces deep syntax trees containing content-word lemmas and syntactic formemes.These trees represent the syntactic shape selected during sentence planning.
- 2 Generator Setting: The resulting deep syntax trees are linearized into strings by the external TectoMT surface realizer.This realizes the two-step pipeline after sentence planning.
- 2 Generator Setting: The joint mode combines sentence planning and surface realization by producing natural-language sentences directly.Unlike the two-step mode, it does not generate an intermediate tree.
- 2 Generator Setting: The two-step mode abstracts complex surface syntax and morphology into a handcrafted module, whereas the joint mode avoids explicit structure and pipeline error accumulation.The two approaches therefore expose different design trade-offs.
3 The Seq2seq Generation Model
The generator encodes dialogue acts, deep syntax trees, and sentences as token sequences for an attention-based seq2seq model. Beam search generates candidates, while a classifier-based reranker penalizes outputs that omit required information or add irrelevant content.
- Input and output representations: DA, deep syntax trees, and sentences are represented as token sequences so recurrent neural-network components can process them.Dialogue acts use type-slot-value triples; trees use bracketed notation, while reranking treats trees as lemma-formeme token sequences.
- Seq2seq generator: The encoder maps an input sequence to hidden states with an LSTM, and an attention-based LSTM decoder generates the output sequence token by token.The decoder is initialized from the encoder’s final hidden state and uses attention-weighted encoder states as context.
- Decoding: Beam search decodes left to right while retaining the top n output sequences ranked by accumulated log probability.Each partial sequence is expanded one token at a time.
- Reranker: The reranker uses an RNN encoder and logistic classifier to predict a 1-hot vector indicating dialogue-act types and slot-value combinations present in an output.Its predictions support penalties for missing required information or adding irrelevant information.
- Generation modes: The model supports both direct string generation and deep syntax-tree generation within the same sequence-based architecture.The paper illustrates tree and string sequences as alternative output representations for the generator and reranker.
4 Experiments
Experiments evaluate the generator on the delexicalized BAGEL restaurant-information dataset using automatic metrics and manually assessed semantic errors. Training uses unaligned meaning-representation and sentence pairs, with separate procedures for the generator and reranker.
- Dataset: The BAGEL dataset contains 202 dialogue acts from the restaurant-information domain, with two natural-language paraphrases per act.The data cover properties including locations, price ranges, and food types; some names and phone numbers are delexicalized.
- Data preparation: The setup learns slot-to-word alignments from data instead of using manual annotations, simplifying preparation but making generation harder.Deep syntax trees are obtained automatically with the Treex NLP toolkit.
- Training: Generator training uses Adam to minimize unweighted sequence cross-entropy across 10 runs with different random initializations.Parameters are selected using validation BLEU, with early stopping based on stalled top-10 validation scores.
- Training: The reranker is trained with Adam and cross-entropy in a single run of up to 100 passes, selecting parameters by validation and training Hamming distance.The validation set receives 10 times more importance in the reported training settings.
- Evaluation: Evaluation reports BLEU, NIST, and manually assessed semantic errors from a sample of 42 output sentences.The two paraphrases for each dialogue act are treated as separate training instances but jointly used as references for BLEU and NIST.
5 Results
The experiments compare direct string generation with deep-syntax-tree generation and evaluate fluency, semantic errors, and automatic scores. The joint setup ultimately performs better with reranking, while the two-step setup produces valid structures and different error patterns.
- The evaluation reports BLEU, NIST, and manually assessed semantic errors on 42 output sentences from two cross-validation runs.The experiments compare these results with previous systems on the same dataset.
- Direct string generation produces fluent domain-style sentences, but greedy decoding frequently makes semantic errors involving confused slot values.Beam search improves BLEU, while reranking reduces semantic errors and substantially improves automatic scores.
- Deep-syntax-tree models learn the domain style and generally produce valid trees, while surface realization introduces few errors.Their remaining errors more often involve missing, incorrect, or repeated information than confusion between semantically similar items.
- The two approaches exhibit different decoding trade-offs: beam search can improve BLEU while increasing semantic errors, and larger beams can lower BLEU despite fewer errors.NIST better reflects the latter trade-off for syntax-generating models.
- With reranking, the joint string-based model produces significantly better BLEU/NIST scores than the tree-based model while reducing semantic errors.The comparison uses the same generator and shows a different outcome from the no-reranker comparison.
6 Related Work
Related work frames pipeline versus joint generation as a recurring architectural choice and positions this paper as a direct comparison using both strings and deep syntax trees. The approach also emphasizes training from unaligned data and an order-of-magnitude smaller dataset than other RNN-based systems.
- Pipeline systems in spoken dialogue generation often emphasize sentence planning through handcrafted generators or perceptron-guided search.The choice between pipeline and joint generation has often depended on architecture or domain.
- Table 2 provides example outputs from different generator setups using beam size 100, with errors marked by type.The marked categories include missing, superfluous, repeated information, and disfluency.
- The generator is trainable from unaligned pairs of meaning representations and sentences alone.This avoids requiring manually annotated alignments between input slots and target words.
- Joint-generation systems learn output production without separating sentence planning from surface realization, while this work extends that comparison to deep syntax trees and strings.The supplied related-work passage identifies the paper as a direct comparison of both approaches.
- The study experiments with an order-of-magnitude smaller dataset than other RNN-based systems.This is presented as an extension of prior RNN-based NLG work.
7 Conclusions and Future Work
The paper concludes that both generation modes can be trained successfully with limited data, but direct joint generation is more favorable overall. It also reports stronger automatic scores, relevant outputs, and planned extensions to other datasets and model components.
- The joint direct-generation setup achieves higher n-gram-based scores than the two-step tree-based setup while producing a similar number of semantic errors.Both approaches use the same sequence-to-sequence generator.
- The generator learns meaningful utterances from much less training data than typically used by RNN-based approaches.The resulting models also produce fluent, coherent sentences and valid bracketed deep syntax trees.
- The generator surpasses the previous best BLEU/NIST scores on the same dataset while reducing irrelevant output information.The comparison is against a perceptron-based generator.
- The authors release the generator on GitHub.
- Future work includes applying the generator to other datasets and improving reranking, encoding, and sequence-level training.The planned changes include a bidirectional encoder and sequence-level training.