Source-linked AI summary
The E2E Dataset: New Challenges For End-to-End Generation
Jekaterina Novikova, Ondřej Dušek, Verena Rieser
TL;DR
Existing end-to-end NLG datasets have limited size and linguistic variety, while richer dialogue and discourse phenomena remain difficult to reproduce. This paper introduces and analyzes the larger crowdsourced E2E restaurant dataset, evaluates a baseline, and finds that its reference diversity creates new generation challenges while still supporting competitive BLEU performance. The dataset is intended to encourage more natural and varied outputs than template-like resources.
Problem
Existing end-to-end NLG datasets are smaller and have limited the reproduction of rich dialogue, discourse, and linguistic phenomena.
Method
The paper constructs and analyzes a 50k-instance crowdsourced restaurant dataset, then evaluates a TGen end-to-end baseline on it.
Results
TGen achieves BLEU 0.6276 on BAGEL and 0.7270 on SFRest as comparison scores, while its E2E development-set BLEU remains in the same range despite greater dataset variety.
Takeaways & Limitations
The E2E data promises more natural and varied outputs than previous template-like datasets.
Takeaways & Limitations
Current end-to-end generators cannot handle out-of-vocabulary words, requiring sparsely occurring string attributes to be delexicalized in the baseline.
Abstract
from arXiv · showhide
This paper describes the E2E data, a new dataset for training end-to-end, data-driven natural language generation systems in the restaurant domain, which is ten times bigger than existing, frequently used datasets in this area. The E2E dataset poses new challenges: (1) its human reference texts show more lexical richness and syntactic variation, including discourse phenomena; (2) generating from this set requires content selection. As such, learning from this dataset promises more natural, varied and less template-like system utterances. We also establish a baseline on this dataset, which illustrates some of the difficulties associated with this data.
1 Introduction
End-to-end NLG methods aim to reduce the need to redevelop systems for each domain, but existing datasets have limited the dialogue and discourse phenomena they can reproduce. The E2E dataset addresses this gap with a larger crowdsourced restaurant-domain resource and a baseline evaluation.
- End-to-end NLG systems jointly learn sentence planning and surface realisation from non-aligned data, supporting faster development across application domains.
- Existing datasets are relatively small and have limited end-to-end systems' ability to reproduce rich dialogue and discourse phenomena.
- 50k instances form the new crowdsourced E2E restaurant-domain dataset for end-to-end NLG research.
- The paper analyzes the dataset's challenges and establishes a baseline using a previous end-to-end approach.
2 The E2E Dataset
The E2E dataset combines pictorial meaning representations with an extensive restaurant-domain ontology and separate train, validation, and test splits. Its instance structure supports varied combinations of restaurant attributes and references.
- The dataset was collected on CrowdFlower and quality-controlled following the procedure of Novikova et al. (2016).
- The dataset contains more than 50k dialogue-act-based MR and reference combinations, averaging 8.1 references per MR.
- Each meaning representation contains 3–8 restaurant attributes, such as name, food, or area, together with their values.
- The data is split into training, validation, and testing sets in a 76.5-8.5-15 ratio, with distinct meaning representations across sets.
3 Challenges
Compared with earlier restaurant-domain datasets, E2E is larger and linguistically richer, while its references introduce content-selection mismatches and more difficult generation conditions.
- Size: The E2E dataset is significantly larger than BAGEL and SFRest in instances, unique MRs, and average references per MR.Its references contain up to six sentences, more slot-value pairs, and longer words and sentences than the other datasets.
- Lexical Richness: E2E references have the highest MSTTR at 0.75, compared with BAGEL's lowest value of 0.41, and contain about 15% more infrequent words than the other datasets.MSTTR measures lexical diversity, while lexical sophistication measures the proportion of words outside the 2,000 most frequent British National Corpus words.
- Lexical Richness: 61% of trigrams and 50% of bigrams occur only once in E2E, whereas neither BAGEL nor SFRest contains singly occurring bigrams or trigrams.This concentration of infrequent phrases creates a challenge for efficient training and contributes to out-of-vocabulary problems for current end-to-end generators.
- Syntactic Variation and Discourse Phenomena: E2E has the lowest proportion of simple references at 46%, while 14% are level-2 sentences and 16% are level-6-or-7 sentences, the highest among compared datasets.The dataset includes coordination, gerund phrases, subordinate clauses, referring expressions, and multiple combined structures.
- Content Selection: 40% of sampled E2E MR-reference pairs contain either additional or omitted information, requiring systems to learn which MR content to realise.Workers could skip unimportant attribute values, and references could add information absent from the MR.
4 Baseline System Performance
The TGen baseline produces fluent, relevant outputs and achieves BLEU performance comparable to results on earlier datasets, but it still requires delexicalization and does not learn content selection.
- TGen uses an attention-based sequence-to-sequence model with beam search and a reranker that penalizes outputs omitting input-MR attributes.Name and near values are delexicalized because TGen handles unknown vocabulary poorly.
- TGen achieves a BLEU score in the same range as its BAGEL and SFRest results despite the E2E dataset’s greater variety.The comparison uses BLEU scores of 0.6276 for BAGEL and 0.7270 for SFRest.
- Cursory checks find outputs mostly fluent and relevant, including long multi-sentence outputs with referring expressions and ellipsis.An example includes two sentences describing a restaurant’s type, food, rating, area, and nearby hotel.
- TGen requires delexicalization and forces verbalization of all MR attributes because it does not learn content selection.This contrasts with E2E references, where 40% of sampled data contains additional or omitted information.
5 Conclusion
The paper presents E2E as a large, openly released restaurant-domain dataset for end-to-end statistical NLG. Its lexical richness, syntactic complexity, discourse phenomena, and content-selection demands are intended to support more natural and varied outputs than template-like datasets.
- The E2E dataset contains more than 50k crowdsourced restaurant-domain instances and is freely released through the E2E NLG Shared Task.The data was collected using pictorial stimuli.
- E2E references exhibit lexical richness, syntactic complexity, and discourse phenomena that create new challenges for end-to-end generation.Generating from the dataset also involves content selection.
- Learning from pictorially elicited E2E data promises more natural and varied outputs than previous template-like datasets.The paper characterizes the expected outputs as less template-like.
- Future work targets greater complexity through comparison, summarization, and recommendation requests, plus dialogue-context data for multi-turn discourse phenomena.These directions are intended to replicate earlier rule-based and statistical approaches and model discourse across turns.
The E2E Dataset Supplementary Material: Baseline Model Parameters
The supplementary material specifies training and decoding settings for TGen’s sequence-to-sequence generator and reranker, including optimization, sequence lengths, validation, and postprocessing choices.
- The main TGen model uses Adam with learning rate 5e-4, LSTM cells, 50-dimensional embeddings, batch size 20, and maximum lengths of 10 input pairs and 80 output tokens.Training runs for up to 20 epochs with 2,000 validation instances reserved.
- The generator tokenizes and lowercases data, replaces name and near values with placeholders, minimizes token-level cross entropy, and uses BLEU-based early stopping.Training stops when the top three BLEU results remain unchanged for five epochs.
- The reranker uses Adam with learning rate 1e-3, 50-dimensional embeddings, batch size 20, maximum input length 80, and 20 training epochs.It reserves 2,000 training instances for validation.
- The reranker classifies attribute-value presence from references to reorder generator outputs according to how well they reflect the input MR.Because name and near are delexicalized, it uses 21 classifiers.
- A high reranking penalty promotes outputs that perfectly cover the input MR, after which delexicalized name and near values are restored by postprocessing.This decoding design prioritizes complete attribute coverage.