PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge

Ondřej Dušek, Jekaterina Novikova, Verena Rieser

arXiv:1901.07931v3cs.CL

TL;DR

The paper addresses how end-to-end NLG should be evaluated when systems must handle richer language phenomena and whether pictorial meaning representations improve data elicitation. It combines analysis of shared-task evaluations with comparisons of representation formats, finding benefits for pictorial MRs while noting limits on the semantic complexity pictures can express.

  • Problem

    Large-scale comparative evaluation is needed to identify future NLG research challenges, including the suitability of richer meaning representations for eliciting data.

  • Method

    The paper analyzes the E2E NLG shared task and evaluates pictorial versus textual meaning representations using natural-language utterances elicited from them.

  • Results

    Pictorial meaning representations elicit texts that are significantly less diverse but significantly more informative, natural, and better phrased than texts elicited from textual representations.

  • Takeaways & Limitations

    Pictorial meaning representations have specific benefits for eliciting NLG data, while the dataset and shared task support broader analysis of end-to-end generation.

  • Takeaways & Limitations

    Pictures have a limit in the meaning complexity they can express, and crowd workers have a limit in the semantic complexity they can handle.

Abstract

from arXiv · show

This paper provides a comprehensive analysis of the first shared task on End-to-End Natural Language Generation (NLG) and identifies avenues for future research based on the results. This shared task aimed to assess whether recent end-to-end NLG systems can generate more complex output by learning from datasets containing higher lexical richness, syntactic complexity and diverse discourse phenomena. Introducing novel automatic and human metrics, we compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -- with the majority implementing sequence-to-sequence models (seq2seq) -- as well as systems based on grammatical rules and templates. Seq2seq-based systems have demonstrated a great potential for NLG in the challenge. We find that seq2seq systems generally score high in terms of word-overlap metrics and human evaluations of naturalness -- with the winning SLUG system (Juraska et al., 2018) being seq2seq-based. However, vanilla seq2seq models often fail to correctly express a given meaning representation if they lack a strong semantic control mechanism applied during decoding. Moreover, seq2seq models can be outperformed by hand-engineered systems in terms of overall quality, as well as complexity, length and diversity of outputs. This research has influenced, inspired and motivated a number of recent studies outwith the original competition, which we also summarise as part of this paper.

1. Introduction

The E2E NLG Challenge introduced a large-scale shared task and extended evaluation of end-to-end systems, examining their ability to handle richer language-generation challenges. The analysis compared techniques, errors, output properties, and evaluation methods to identify strengths, weaknesses, and future research directions.

  • Analysis and future work: The paper provides detailed error and output-similarity analyses, identifies system strengths and weaknesses, and discusses future challenges for E2E generation and NLG evaluation.It also releases participating outputs and human ratings to support further research on automatic quality estimation.
  • Challenge and dataset: The challenge evaluated end-to-end NLG systems on a new restaurant-domain dataset designed to introduce open vocabulary, complex syntax, and diverse discourse phenomena.The dataset contained 50k instances and aimed to test whether richer training data enabled more complex outputs.
  • Challenge and dataset: 62 submissions from 17 institutions enabled a large-scale comparison of current end-to-end generation approaches.About one-third of submissions came from industry, and the participation level supported broader conclusions about the task.
  • Evaluation and analysis: The paper compares submitted systems with a challenging baseline using automatic metrics, novel text-based measures, and human evaluation.The expanded evaluation accounts for output complexity, diversity, semantic accuracy, and naturalness.
  • Results: Seq2seq systems generally score highly on word-overlap measures and human naturalness rankings, whereas manually engineered systems outperform some seq2seq systems in overall quality, diversity, and complexity.The results reveal a discrepancy between data-driven seq2seq models and template- or rule-based systems.

2. Domain and Task

The task treats NLG as converting a meaning representation into one or more natural-language sentences. End-to-end systems learn this conversion from MR–reference pairs without fine-grained alignments between MR elements and words.

  • Task definition: NLG converts an input meaning representation into a natural-language utterance consisting of one or more sentences.The paper focuses on end-to-end data-driven generation from simple MR and reference-text pairs.
  • Task definition: End-to-end generators are trained without fine-grained alignments between meaning-representation elements and words.The approach relies on paired meaning representations and reference texts rather than explicit element-to-word alignments.
  • Task definition: The E2E dataset illustrates the task with structured attributes such as name, price range, and customer rating paired with a natural-language description.For example, an MR describing The Wrestlers as cheap and poorly rated is paired with a sentence expressing those properties.

3. Data Collection Procedure

The data collection procedure used crowdsourcing with quality controls and compared textual with pictorial meaning representations. Pictorial representations produced more varied, informative, natural, and better-phrased texts, but introduced noise and were costly to scale.

  • Worker selection: Crowd workers were recruited through CrowdFlower and screened for English proficiency, minimum completion time, and duplicate submissions.The procedure also removed contributors who completed tasks too quickly and checked that utterances were well-formed English sentences using three JavaScript validators.
  • Motivation: The study crowdsourced natural-language paraphrases for meaning representations to obtain sufficient high-quality NLG training data.The authors identified data quality and the kinds of meaning representations that elicit spontaneous, natural, varied language as central challenges.
  • Results: Collection time did not differ significantly between modalities, while reference length depended mainly on the number of MR attributes rather than whether text or pictures were used.This comparison considered both character count and number of sentences.
  • Results: Compared with textual MRs, pictorial MRs elicited significantly less semantically similar texts, significantly more informative texts, and significantly more natural and better-phrased utterances.The naturalness and phrasing difference was significant at p < 0.001, while the informativeness difference was significant at p < 0.01.
  • Results: Pictorial MRs introduced more omissions and noise, had limited semantic expressiveness, and required an expensive semi-automatic production process, so they were used for 20% of the full dataset.The eatType = restaurant attribute was particularly difficult to visualise and was often omitted by workers.

4. The E2E NLG dataset

The E2E dataset is a large restaurant-domain resource designed to provide greater lexical richness, syntactic complexity, and discourse variation than earlier datasets. Its size, diversity, and attribute complexity make it challenging for end-to-end data-driven NLG.

  • 4. The E2E NLG dataset: 50k instances make the E2E dataset substantially larger than previous restaurant-domain NLG datasets, including BAGEL and SFRest.
  • 4.1. Size: E2E references average 8.27 references per MR, compared with fewer than two for the other datasets.
  • 4.1. Size: E2E meaning representations are more complex, averaging twice as many slot-value pairs as SFRest and slightly more than BAGEL.
  • 4.2. Lexical Richness: E2E has a vocabulary more than twice the size of SFRest and over five times larger after delexicalisation.
  • 4.2. Lexical Richness: The E2E dataset has higher MSTTR than the other sets, at 0.71 versus at most 0.65, indicating greater lexical diversity.
  • 4.4. Attribute Coverage: Roughly 40% of sampled E2E references omit at least one MR attribute, often eatType=restaurant.

5. Systems in the Competition

The competition compared diverse NLG architectures, with seq2seq systems forming the majority of primary entries alongside data-driven, rule-based, and template-based systems. A TGen seq2seq model provided a strong baseline for the challenge.

  • 5. Systems in the Competition: The challenge received 62 submissions from 17 institutions in 11 countries, with about one-third submitted by industry.
  • 5. Systems in the Competition: The evaluation focused on 20 primary systems from 14 groups, comparing them with the TGen baseline using automatic metrics and human judges.
  • 5. Systems in the Competition: TGen combines an attention-based encoder-decoder RNN with beam search and an LSTM reranker that penalises outputs missing input attributes.
  • 5. Systems in the Competition: TGen appeared to be a strong baseline, generating fluent and relevant outputs in most cases.
  • 5. Systems in the Competition: Seq2seq systems were the largest group, accounting for 12 of 20 primary systems, and all used beam search.
  • 5. Systems in the Competition: The competition also included fully trainable non-seq2seq systems, rule-based pipelines, and template-filling systems.

6. Addressing the Challenges

The competition systems addressed open vocabulary, semantic coverage, and output diversity through delexicalisation, copying, reranking, attention modifications, data augmentation, and controlled generation. These mechanisms targeted the complexity of the E2E dataset and the weaknesses of unconstrained seq2seq generation.

  • 6.1. Open Vocabulary: All challenge systems addressed open vocabulary, with data-driven systems mostly using partial delexicalisation and rule-based systems using full delexicalisation.
  • 6.1. Open Vocabulary: Alternative open-vocabulary strategies included copy mechanisms, sub-word units, and character-level generation.
  • 6.2. Semantic Control: Most systems therefore used semantic coverage mechanisms such as rerankers, aligners, attention penalties, attention memory, or explicit slot classifiers.
  • 6.2. Semantic Control: Vanilla seq2seq systems cannot control whether all input slots are realised because they are not explicitly trained to attend once to each slot.
  • 6.3. Output Diversity: Data augmentation strategies included Personage-generated outputs, slot reshuffling, and aligned sentence fragments to expand training data.
  • 6.3. Output Diversity: Systems increased diversity through complex-sentence filtering, diverse ensembling, first-word control, and enriched meaning representations.
  • 6.4. Systems outside the competition and E2E-inspired work: The paper surveys post-challenge approaches because solving these challenges remains an ongoing effort, including further delexicalisation and copy mechanisms.

7. Evaluation Setup

The evaluation combined automatic measures of reference similarity, textual complexity, and diversity with large-scale human assessment. It also introduced RankME and TrueSkill-based procedures to compare many systems efficiently.

  • 7. Evaluation Setup: Systems were evaluated with automatic metrics and a large-scale crowdsourced human evaluation.
  • 7. Evaluation Setup: Automatic evaluation used word-overlap measures for reference similarity and textual measures for output complexity and diversity.
  • 7. Evaluation Setup: The evaluation motivates caution about relying only on automatic metrics because they may correlate weakly with human judgements.
  • 7. Evaluation Setup: The paper extended Rank-based Magnitude Estimation and verified it at scale for comparing NLG systems.
  • 7. Evaluation Setup: TrueSkill was introduced to NLG to compute an overall ranking through direct system comparisons.

Word-overlap metrics

The challenge evaluated system outputs against human references using several word-overlap metrics, including BLEU, NIST, METEOR, ROUGE-L, and CIDEr.

  • BLEU measures harmonic mean n-gram precision against human references, with a brevity penalty for shorter outputs.
  • NIST extends BLEU by weighting less frequent n-grams more heavily and using a different length penalty.
  • METEOR combines unigram precision and recall, using exact matches plus stemming- and WordNet-based fuzzy matching.
  • ROUGE-L computes an F-measure from longest common subsequences between system outputs and human references.
  • CIDEr averages TF-IDF-weighted n-gram cosine similarities between system outputs and reference sentences.

Textual metrics

The evaluation combines lexical, syntactic, diversity, and human-rating measures to assess system outputs and compare them with dataset characteristics.

  • Lexical sophistication, MSTTR, and revised D-level metrics measure lexical richness and syntactic complexity in system outputs.These metrics also allow comparison with the training and test sets.
  • Restaurant names were delexicalized before textual scoring to focus metrics on system-produced style.
  • Rank-based Magnitude Estimation collected relative quality and naturalness judgments from crowdsourced comparisons.Workers ranked five outputs generated for one meaning representation, with ties permitted.
  • TrueSkill aggregated pairwise comparisons into Bayesian system rankings while reducing the required comparison data.Earlier work showed that the number of comparisons could be reduced without compromising final rankings.
  • Informativeness was not evaluated separately because training instances do not always verbalize every meaning-representation attribute.The study therefore collected separate ranks for quality and naturalness.

8. Results

The results reveal a trade-off across word overlap, naturalness, semantic accuracy, complexity, and diversity: seq2seq systems often sound natural, while controlled or engineered systems better match richer outputs.

  • Seq2seq systems generally achieve the strongest word-overlap scores and naturalness ratings, while hand-engineered systems can attain higher overall quality.Slug wins quality, and Sheff2 wins naturalness, but quality rankings include multiple architecture types.
  • Seq2seq outputs are usually syntactically simpler than human-authored data and hand-engineered outputs unless diversity control is applied.Most seq2seq systems produce more low-level D-level sentences and fewer high-level ones than other architectures.
  • None of the systems matches human references in overall lexical sophistication or diversity.Adapt approaches human lexical sophistication, while Adapt and TR2 are among the closest systems on diversity dimensions.
  • Template mining and explicit diversity control produce the most diverse outputs, whereas vanilla seq2seq and handcrafted templates tend toward low diversity.TR2 and FORGe3 show that mined templates can generate diverse outputs without concentrated diversity engineering.
  • Strong semantic control during decoding is associated with better meaning-representation coverage in seq2seq systems.Systems without reranking or comparable control tend to rank near the bottom, while several controlled systems report low estimated SER.
  • Missing information is the most common semantic error, while hallucinated information is less frequent but still serious.Systems with fewer hallucinations tend also to miss less information, and diversity attempts may reduce semantic accuracy.

9. Conclusion

The first E2E NLG shared task established a large benchmark, evaluated diverse architectures with automatic and human measures, and identified design trade-offs for future systems.

  • The E2E dataset is a substantially larger restaurant-domain benchmark designed to support NLG evaluation at greater scale.The dataset is described as an order-of-magnitude larger than related datasets.
  • The challenge compared 62 submissions from 17 institutions, spanning seq2seq, other data-driven, rule-based, and template-based systems.Twenty primary submissions also underwent crowdsourced human evaluation.
  • Seq2seq systems perform strongly on word overlap and naturalness, but other architectures can achieve higher overall quality, complexity, length, and diversity.The winning Slug system is seq2seq-based, while hand-engineered systems often produce more complex and diverse outputs.
  • Vanilla seq2seq models tend to produce short, low-diversity, low-complexity outputs that can become repetitive.
  • Strong semantic control during decoding is crucial for preserving input meaning, because missing and added information remain serious errors.
  • Improving diversity remains difficult because it can lower semantic accuracy or naturalness.

Appendix A. Detailed Results

Appendix A documents detailed comparisons among E2E challenge systems using automatic metrics, syntactic-complexity tests, inter-system similarity measures, and correlations with human judgments.

  • Automatic evaluation: Table A.14 lists E2E challenge submissions with automatic metric scores and a normalized average across metrics.The table identifies primary systems and reports normalized aggregate scores.
  • Automatic evaluation: Figure A.6 compares system-output similarity using BLEU, NIST, METEOR, ROUGE-L, and CIDEr under alternative reference settings.The comparisons include another system, the full test set with multiple human references, and a randomly chosen single human reference.
  • Output complexity: Figure A.7 tests whether systems differ significantly in syntactic complexity using Kolmogorov-Smirnov tests for discrete distributions.Bright colors indicate statistically significant differences at p < 0.05.
  • System comparisons: Figure A.8 represents similarities between systems with Bhattacharyya distance, where darker colors indicate greater distance.The visualization encodes how different systems are from one another.
  • Human evaluation: Tables A.15 and A.16 report Pearson correlations between automatic metrics and human scores of naturalness and quality.Asterisks mark statistical significance at p < 0.05, while bold indicates the highest correlation value.
Loading 1901.07931v3…