Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge

• The report extends prior work with corrected dataset evaluation, novel complexity, diversity, and semantic metrics, output-similarity and error analyses, future directions, and released test outputs with human ratings.It also releases 36k system outputs paired with user ratings for research on automatic quality estimation.Source: §1 · pp. 2, 4
• The challenge targets data-driven systems that jointly learn sentence planning and surface realization, reducing annotation needs by avoiding costly semantic alignment while training on parallel data.Source: §1 · pp. 2–3
• The 50k-instance restaurant dataset introduces open vocabulary, complex syntax, and diverse discourse phenomena, addressing limits of earlier end-to-end datasets.Source: §1 · p. 3
• The challenge aimed to test whether richer training data enables more complex outputs, attracting 62 submissions from 17 institutions in 11 countries, about one-third from industry.Source: §1 · p. 3
• The paper presents the first at-scale human assessment of novel end-to-end generation, evaluating submissions against a challenging baseline with automatic and human measures.Source: §1 · pp. 3–4
• Seq2seq systems generally score high on word-overlap measures and human naturalness rankings, while manually engineered systems outperform some seq2seq systems in overall quality, diversity, and output complexity.Source: §1 · p. 4

2. Domain and Task

• The task is to generate one or more natural-language sentences from an MR, training on simple MR–reference-text pairs without fine-grained alignments.Source: §2 · p. 4

3. Data Collection Procedure

• To build sufficiently large, high-quality training data, the researchers crowdsourced MR paraphrases and ran a 75-MR pilot to investigate data quality and elicitation.The pilot used MRs with three, five, or eight attributes and tested automatic validation and pictorial representations before applying successful techniques to the full dataset.Source: §3 · pp. 5–6

3.1. Automatic Validation Measures

• Workers were screened for English nativeness and location and required to spend at least 20 seconds per page; validators checked legal characters, minimum length, and duplicate submissions.Manual checks of initial trials found that these validation methods rejected 100% of bad submissions.Source: §3.1 · pp. 7–8

3.2. Meaning Representations: Pictures and Text

• Textual MRs were randomized lists of attribute-value pairs, while pictorial MRs combined attribute icons with a city map to represent area and proximity.The study explored these modalities because textual/logical descriptions can limit semantic complexity and prime workers’ lexical choices.Source: §3.2 · p. 8

3.3. Data Collection Setup

• Using CrowdFlower with automatic checks and textual or pictorial MRs, the pre-study collected 1,133 utterances from 75 MRs (744 textual, 498 pictorial); the same setup later collected the whole E2E NLG dataset.Source: §3.3 · pp. 8–9

3.4. Results and Discussion

• Collection time did not differ significantly, and utterance length depended chiefly on MR attribute count; pictorial MRs also increased omission noise and were costly to produce.The team therefore used pictorial MRs for 20% of the full dataset, textual MRs for the rest, and collected multiple references per MR to increase quality and diversity.Source: §3.4 · pp. 10–11
• Pictorial MRs elicited texts rated significantly more informative (p < 0.01), natural, and better phrased (p < 0.001), but less semantically similar to the underlying MR.The authors suggest reduced lexical priming may explain the human-evaluation gains.Source: §3.4 · p. 10

4. The E2E NLG dataset

• The crowdsourced dataset contains 50k restaurant-domain instances and was designed to offer greater lexical richness, syntactic complexity, and discourse diversity than earlier datasets (Novikova et al., 2017b).For fair comparison, the study examines lexicalised and delexicalised versions of E2E, BAGEL, and SFRest, with SFRest also compared in an inform-only subset.Source: §4 · pp. 11–13

4.1. Size

• E2E contains over 20 times SFRest’s human-reference text and averages 8.27 references per MR, versus fewer than two for other sets.It also has more distinct MRs and instances. Its 82:9:9 train/validation/test split holds out unseen MRs; test MRs are somewhat longer and more complex than training MRs.Source: §4.1 · pp. 13, 15, Table 2 · p. 14

4.2. Lexical Richness

• E2E has a vocabulary 2x SFRest’s (over 5x after delexicalisation) and nearly 10x its distinct trigrams (over 13x delexicalised).The larger vocabulary and n-gram counts make data-driven generation challenging even within this narrow domain.Source: §4.2 · p. 15, Table 4 · p. 16
• E2E’s MSTTR is 0.71 versus at most 0.65 for other sets; delexicalised E2E scores 0.66 versus 0.55 for SFRest-inf and 0.48 for BAGEL.E2E is more lexically sophisticated than SFRest but slightly less than BAGEL; after delexicalisation, it is almost twice as sophisticated as both.Source: Table 4 · p. 16, §4.2 · p. 17
• Delexicalised E2E has the highest entropy across n-grams, including trigram entropy of 12.1 versus SFRest’s 10.5 and next-word conditional entropy of 2.446 versus 1.414.Before delexicalisation, E2E’s unigram and bigram entropy is slightly lower than SFRest’s, while its trigram entropy is higher than any other set.Source: Table 4 · p. 16, §4.2 · pp. 17–18

4.3. Syntactic Complexity

• E2E references are more syntactically complex on average: mean D-Level is 2.17, compared with 1.32 for BAGEL and 1.25 for SFRest.Simple level-0/1 sentences account for 46% of E2E versus 59–66% elsewhere; E2E also has more level-2 sentences (14%) and level-6/7 sentences (15%, versus 7% for BAGEL and 8% for SFRest).Source: Figure 4 · p. 18, §4.3 · pp. 19–20

4.4. Attribute Coverage

• Roughly 40% of sampled E2E MR-reference pairs contain additional or omitted information, so multiple references per MR help represent this variation.Workers were asked to verbalise all MR information but were not penalised for omissions; variation often involves the eatType=restaurant attribute.Source: Table 5 · p. 20, §4.4 · pp. 20–21

4.5. Following-on Datasets

• The E2E dataset inspired extensions for studying style, sentence planning, dialogue-act annotation, open vocabulary, and web-based restaurant data.These include work by Juraska and Walker (2018), Oraby et al. (2018a,c), Reed et al. (2018), Balakrishnan et al. (2019), Roberti et al. (2019), and Oraby et al. (2019).Source: §4.5 · p. 21

5. Systems in the Competition

• 62 systems from 17 institutions entered the challenge; 20 primary systems from 14 groups were evaluated with automatic metrics and human judges.About one-third of submissions came from industry; two industry groups withdrew and one group requested anonymisation after receiving automatic evaluation results.Source: §5 · p. 23

5.1. Baseline System

• TGen established the baseline with an attention-based seq2seq model, beam search, and an LSTM reranker that penalises outputs omitting input attributes.The model delexicalises the sparse attributes name and near; development-set metrics and cursory manual checks indicated fluent, relevant outputs in most cases.Source: §5.1 · pp. 23–24

5.2. Seq2seq-based systems

• Seq2seq was the largest group, with 12 of 20 primary systems using beam search and many adding mechanisms for coverage, diversity, or other architectural enhancements.Variants included TGen-based data augmentation and REINFORCE fine-tuning, pointer networks, and custom encoders and decoders.Source: §5.2 · pp. 24–25

5.3. Other data-driven systems

• Other trainable systems included SC-LSTM language models with differing semantic control and Sheff1, a non-neural imitation-learning system that selected slots and generated their wording.Sheff1’s classifiers were trained with LOLS to optimise BLEU, ROUGE-L, and slot error.Source: §5.3 · p. 25

5.4. Rule-based systems

• The rule-based entries were DANGNT, which selected phrases before lexicalising slot values, and FORGe1, which used Meaning-Text Theory grammars, per-slot templates, aggregation, and surface realisation.Source: §5.4 · pp. 25–26

5.5. Template-based systems

• Template-based entries FORGe3 and TR2 mined templates by delexicalising slot values, while TUDA used manually designed templates informed by, but not directly derived from, the dataset.Source: §5.5 · p. 26

6.1. Open Vocabulary

• Template- and rule-based systems fully delexicalise slot values, most data-driven systems delexicalise name and near, and some seq2seq systems instead use copying, subwords, or characters.TR1 also delexicalises priceRange and customerRating; Adapt, Chen, and Harv use copy mechanisms, Zhang uses byte-pair subwords, and NLE works at character level.Source: §6.1 · pp. 26–27

6.2. Semantic Control

• Vanilla seq2seq systems cannot ensure all input slots are realised, so most add semantic checks; MR-classification reranking is the most frequent approach among E2E submissions.Generators produce beam-search candidates, then penalize missing or deviating slots; TGen-based systems and NLE use classifiers, while Slug systems use a heuristic aligner and Harv uses attention weights.Source: §6.2 · pp. 27–28
• Some systems modify attention directly, while Adapt, TR1, and Sheff2 use no explicit semantic control; non-seq2seq systems use slot-tracking gates, classifiers, or slot-level planning.Chen adds cumulative attention memory and Zhang regularizes attention; ZHAW1 and ZHAW2 use SC-LSTM gates, and Sheff1 predicts which slot to verbalise next.Source: §6.2 · p. 28

6.3. Data Augmentation and Diversity

• Challenge systems augment or filter training data to alter quality, robustness, complexity, or diversity, including Personage outputs, shuffled slots, aligned sentence fragments, and structurally grouped ensembles.Sheff1 selects one high-frequency reference per MR to stabilise training, while Slug-alt trains only on complex sentences for more sophisticated outputs.Source: §6.3 · pp. 28–29
• ZHAW1 and ZHAW2 sample outputs generated from different first words, while Adapt preprocesses MRs to specify words for the main generator.ZHAW1 samples only semantically correct outputs.Source: §6.3 · p. 29

6.4. Systems outside the competition and E2E-inspired work

• Outside the challenge, open-vocabulary work uses copy mechanisms, copy-plus-delexicalisation, subword denoising autoencoders, and character-based models (Freitag and Roy, 2018; Roberti et al., 2019).Roberti et al. (2019) also extend the E2E dataset with more restaurant names.Source: §6.4 · pp. 29–30
• Related semantic-coverage work explores checklist decoding, multi-level attention and gating, multi-level decoders, multi-objective optimisation, and added supervision or decoding constraints.Examples include checklist models, two-level attention (Tran et al., 2017), and E2E work adding sentence-count, unlabelled-data, or linguistic-structure signals.Source: §6.4 · p. 30
• Other diversity research uses latent templates, stylistic or contrast-marker supervision, personality annotations, syntactic features, character-level models, sentence plans, or dialogue context.These approaches include work on E2E data and other domains, with character-level seq2seq models reported to produce greater diversity than word-level models (Jagfeld et al., 2018).Source: §6.4 · pp. 30–31

7. Evaluation Setup

• Because automatic metrics may weakly reflect human judgments, the evaluation paired them with a large-scale crowdsourced human evaluation.The automatic evaluation included a novel application of textual measures and a novel use of word-overlap metrics to assess similarity among systems.Source: §7 · p. 31
• Rank-based Magnitude Estimation (RankME) extended prior work (Novikova et al., 2018) and was verified at scale for comparing many systems.Source: §7 · p. 32
• TrueSkill enabled a data-efficient overall ranking by direct system comparisons rather than costly individual assessments.The paper introduced this algorithm to NLG as a way to compare a large number of systems efficiently.Source: §7 · p. 32

7.1. Automatic Metrics

• Automatic metrics assessed either similarity to corpus references with word-overlap measures or the complexity and diversity of system outputs with textual measures.Source: §7.1 · p. 32

Word-overlap metrics

• BLEU averages clipped n-gram precisions for n=1–4 and applies a brevity penalty when output is shorter than references.A matched n-gram cannot count more times than it appears in any single reference.Source: Word-overlap metrics · p. 32
• NIST weights less frequent, more informative n-grams more heavily than BLEU and uses n=1–5 with a different length penalty.Source: Word-overlap metrics · p. 33
• METEOR measures unigram precision and recall using exact, stemmed, and WordNet-synonym matches, selecting the best-matching reference.Source: Word-overlap metrics · p. 33
• ROUGE-L scores longest common subsequences with an F-measure based on maximum precision and recall across references.A common subsequence preserves word order while allowing intervening words in either sequence.Source: Word-overlap metrics · p. 33
• CIDEr averages cosine similarity over 1–4-grams weighted by TF-IDF, adapting a metric designed primarily for image captions to NLG.Source: Word-overlap metrics · p. 33
• The challenge applied these metrics both against human-authored references and to compare each system’s outputs with those of other systems.Source: Word-overlap metrics · pp. 33–34

Textual metrics

• Textual metrics measured lexical richness and syntactic complexity, with restaurant names delexicalized to prevent infrequent nouns from skewing style scores.The metrics also enabled comparison of output characteristics with the training and test sets.Source: Textual metrics · p. 34

7.2. Human Evaluation

• Crowd workers ranked five randomly selected system outputs for one meaning representation using RankME, with ties permitted.The human evaluation covered the 20 primary systems and the baseline.Source: §7.2 · pp. 34–35
• TrueSkill updated system capability estimates from pairwise output comparisons to produce rankings with fewer direct comparisons.The rankings were run 200 times, and systems whose rank ranges overlapped were grouped into significance clusters.Source: §7.2 · p. 35
• Because training instances did not always verbalize every meaning-representation attribute, the evaluation omitted informativeness and collected separate quality and naturalness rankings.Quality judgments showed the meaning representation; naturalness judgments did not.Source: §7.2 · p. 36

8.1. Word-overlap Metrics

• No primary system beat TGen across all word-overlap metrics or their normalized mean, though seq2seq systems generally had the strongest word-based scores and Slug came close.Several systems beat TGen on individual metrics but not others; automatic-metric results differed from human evaluations.Source: §8.1 · pp. 37–38, Table 8 · p. 37

8.2. Textual Metrics

• Seq2seq outputs were generally less syntactically complex than outputs from other architectures, and FORGe3 alone had a D-level distribution not significantly different from a random test-set reference.Systems varied in syntactic complexity, and few matched the training and test data distributions.Source: §8.2 · p. 38
• No system matched human references in lexical sophistication; Adapt came closest, while MSTTR favored template- or rule-based systems and data-driven systems explicitly targeting diversity.MSTTR is typically higher for longer outputs, including many rule- and template-based systems, potentially because it uses a fixed 50-token window.Source: §8.2 · pp. 39–40
• Most systems produced outputs similar in length to human references, but rule- and template-based systems tended to be more verbose, while Zhang, Sheff2, and Chen were much shorter.The short outputs suggest those systems might fail to verbalize all information in the meaning representation.Source: §8.2 · pp. 40–41
• No system matched the diversity of a randomly selected human reference; template mining and explicit diversity controls produced the most diverse outputs, while vanilla seq2seq and handcrafted templates produced the least.TR2 was closest in entropy and next-word conditional entropy, while Adapt came close in vocabulary size.Source: §8.2 · pp. 41–42

8.3. System Output Similarity

• Seq2seq systems’ outputs were generally most similar to one another, whereas diversity-seeking systems and rule- or template-based systems were less similar to other outputs.Most system outputs, especially data-driven ones, were more similar to one another than to a randomly selected human reference.Source: §8.3 · pp. 42, 44, Figure 5 · p. 43

8.4. Results of Human Evaluation

• Slug ranked highest for quality and Sheff2 for naturalness; quality results spanned architectures, while better-ranking seq2seq systems applied explicit semantic control during decoding.The bottom quality clusters also included seq2seq systems without that mechanism, including Sheff2, despite its top naturalness ranking.Source: §8.4 · pp. 44, 48, Table 11 · p. 45

8.5. Error Analysis: Input MR Coverage

• Template- and rule-based systems mostly scored highly on input-MR coverage, while seq2seq systems without reranking tended to rank near the bottom in human and automatic evaluations.Sheff1, Gong, and Slug had estimated SER below 1.5%; TUDA received a perfect automatic coverage score, partly associated with its low diversity. The pattern-matching coverage and SER estimates may miss valid expressions.Source: §8.5 · pp. 49–50
• Missing information was more common than hallucinated information, though hallucinations remained serious; systems that hallucinated less often tended to miss more information.Source: §8.5 · p. 50
• Diversity attempts could reduce semantic accuracy: Adapt lacked explicit control, FORGe3 had noisy mined templates, and Harv’s reranking was probably less aggressive.ZHAW1, ZHAW2, and especially Slug-alt maintained good coverage alongside diverse outputs through powerful semantic control mechanisms.Source: §8.5 · p. 50

8.6. Winning System

• Slug (Juraska et al., 2018) was the overall winner, combining strong human naturalness and quality ratings and word-overlap scores with top-tier MR coverage, but relatively low output diversity.Its heuristic slot aligner and partially aligned training examples improved coverage over vanilla seq2seq systems.Source: §8.6 · pp. 50–51
• Sheff2 won naturalness but ranked 20th–21st in quality after missing MR content; TGen led average word-overlap metrics and placed in second clusters for quality and naturalness, with middling diversity.TGen also performed well, though not perfectly, in MR coverage evaluations.Source: §8.6 · p. 51

8.7. Lessons Learnt and Future Directions

• For seq2seq systems, strong semantic control is crucial: MR-classification reranking or heuristic alignments worked well, whereas attention-only models performed poorly.The paper says correct semantics should be prioritized over fluency.Source: §8.7 · p. 51
• For limited domains, delexicalizing open-set attributes remained the best approach, while character-level models and copy mechanisms were viable alternatives.The authors attribute low results for Chen, Zhang, and Adapt to weaker semantic control rather than open-vocabulary handling.Source: §8.7 · p. 52
• Hand-engineered systems generally outperformed neural systems in complexity and diversity; mined templates and explicit diversity mechanisms produced varied outputs, while vanilla seq2seq was least diverse.Vanilla seq2seq tends toward frequent phrasing; diversity can improve through data selection, diverse ensembling, or beam sampling, but requires strong semantic control to maintain MR coverage.Source: §8.7 · p. 52
• Rule-based methods worked well in this limited domain, while seq2seq with semantic reranking emerged as the best data-driven option when paired with diversity control and copy mechanisms.Low-effort handcrafted systems could produce correct but repetitive outputs; the comparisons are limited to one domain and are not strictly controlled.Source: §8.7 · pp. 51–52

Conclusion

• The shared task tested learning from meaning-representation/text pairs without fine-grained alignments, using a restaurant dataset an order of magnitude larger than prior public task-oriented NLG data; 62 submissions from 17 institutions were evaluated with five metrics, and 20 received crowdsourced evaluation.The evaluation also compared textual complexity and diversity, and the challenge provided TGen as a baseline.Source: §9 · pp. 52–53
• The challenge has influenced, inspired, and motivated recent NLG studies beyond the original competition.Source: §9 · p. 53
• Seq2seq systems, including winning Slug, tend to score highly on word-overlap and naturalness, whereas hand-engineered systems can achieve higher overall quality, complexity, and diversity but lower naturalness.The conclusion also frames the contrast as an open question about the efficiency, cost, and performance of purely data-driven versus carefully hand-engineered NLG.Source: §9 · pp. 53–54
• Vanilla seq2seq models produce short, repetitive, low-complexity outputs; strong decoding-time semantic control is crucial to prevent omissions or hallucinations, while diversity improvements can reduce accuracy or naturalness.Omissions are the most common semantic error, and both omissions and added information can have serious consequences in task-based dialogue, depending on the application domain.Source: §9 · pp. 53–54
• The challenge released its training dataset, TGen development outputs, participant and baseline test outputs, and corresponding human RankME quality and naturalness ratings.These resources were made available to facilitate further research.Source: §9 · p. 54
• The challenge also released word-overlap metrics and scripts for running the other automatic textual metrics used in its evaluation.Source: §9 · p. 54
• Future work will investigate post-edit and extrinsic evaluations, including contribution to task success, and continue automatic quality estimation using the challenge’s larger data resource.The authors note that the challenge data supports a wider range of quality-estimation experiments than was previously possible.Source: §9 · p. 55

Appendix A. Detailed Results

• Table A.14 lists automatic metric scores for E2E challenge submissions, marking primary systems and reporting a normalized average across metrics.The caption refers readers to Table 8 for the normalization method.Source: Table A.14 · p. 77
• Figure A.6 compares system-output similarity using BLEU, NIST, METEOR, ROUGE-L, and CIDEr, with full-test-set and single-human-reference comparisons.Systems are ordered by architecture in the figure.Source: Figure A.6 · p. 78
• Figure A.7 evaluates significance differences between systems in output syntactic complexity using Kolmogorov-Smirnov test p-values.Bright colors indicate statistically significant differences at p < 0.05.Source: Figure A.7 · p. 79
• Figure A.8 depicts between-system similarities calculated with Bhattacharyya distance, where darker colors indicate greater distance.Source: Figure A.8 · p. 79
• Tables A.15 and A.16 report Pearson correlations between automatic metrics and human scores for naturalness and quality, respectively.Asterisks mark significance at p < 0.05, and bold identifies the highest value.Source: Table A.15 · p. 80, Table A.16 · p. 80