Source-linked AI summary
Findings of the E2E NLG Challenge
Ondřej Dušek, Jekaterina Novikova, Verena Rieser
TL;DR
End-to-end NLG systems have so far been evaluated on small, delexicalised datasets, leaving their performance on richer, more complex data an open question. This paper presents a shared task and dataset to assess them at scale. SLUG is the overall winner, while results also show seq2seq systems are capable but need reranking for high-quality results.
Problem
End-to-end NLG approaches were limited to small, delexicalised datasets, leaving their performance on a larger, more complex dataset to be assessed.
Source: §1 · p. 1Method
The paper presents a shared task using a new crowdsourced dataset and compares participating systems with automatic metrics and human evaluation.
Source: §1 · p. 1, §4.2 · p. 4, §5 · p. 4Results
SLUG was the overall winner, scoring best in human quality evaluation, ranking in the second-best naturalness cluster, and reaching high automatic scores.
Source: §5 · p. 5Takeaways & Limitations
Seq2seq systems are capable but require reranking for high-quality results, while rule-based systems can perform comparably or better in human evaluation despite not beating data-driven systems on automatic metrics.
Source: §5 · p. 5A detailed analysis of results and release of system outputs with user ratings were still being prepared.
Source: §5 · p. 5
Abstract
from arXiv · showhide
This paper summarises the experimental setup and results of the first shared task on end-to-end (E2E) natural language generation (NLG) in spoken dialogue systems. Recent end-to-end generation systems are promising since they reduce the need for data annotation. However, they are currently limited to small, delexicalised datasets. The E2E NLG shared task aims to assess whether these novel approaches can generate better-quality output by learning from a dataset containing higher lexical richness, syntactic complexity and diverse discourse phenomena. We compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -- with the majority implementing sequence-to-sequence models (seq2seq) -- as well as systems based on grammatical rules and templates.
1 Introduction
The paper introduces the first shared task evaluating end-to-end NLG systems for spoken dialogue. It tests these systems at scale on a larger, more complex dataset and compares them with a challenging baseline using automatic and human evaluation.
1 Introduction
- The challenge evaluates end-to-end NLG at scale with human assessment, extending earlier comparisons of only a limited number of approaches.It uses a new crowdsourced restaurant-domain dataset of 50k instances, about 10 times larger and more complex than previous datasets.Source: §1 · p. 1
- The shared task received 62 system submissions from 17 institutions, with about one-third coming from industry.The paper reports automatic and human evaluations against a challenging baseline.Source: §1 · p. 1
2 The E2E NLG dataset
The E2E dataset was collected from textual/logical and pictorial meaning representations, with workers producing multiple references per representation. Its scale and linguistic complexity make it a larger, more challenging resource than comparable earlier datasets.
2.1 Data Collection Procedure
- The dataset combines textual/logical attribute-value sequences in random order with pictorial meaning representations using attribute icons on a city map.The pictorial format represents area and near attributes; textual/logical representations are sequences of attribute-value pairs.Source: §2.1 · p. 2
- Pictorial meaning representations were used for 20% of collection because they produced more varied utterances but also introduced noise.A pre-study found similar collection speed and utterance length, and less lexical priming; pictorial-MR utterances were judged more informative, natural, and better phrased.Source: §2.1 · p. 2
- Workers were not penalised for omitting attributes, adding training-data noise, while multiple human references per meaning representation support data-driven approaches.Workers were asked to verbalise all information, but skipping an attribute was permitted.Source: §2.1 · p. 2
2.2 Data Statistics
- Over 50k references cover 6k distinct meaning representations, averaging 8.27 human references per representation.The dataset is 10 times bigger than previous sets in comparable domains; references can contain up to six sentences and meaning representations contain more attributes.Source: §2.2 · p. 2
3 Systems in the Competition
The competition drew broad participation and selected primary systems for evaluation against the TGEN baseline. The participating approaches ranged from data-driven models to rule-based and template-based systems.
3 Systems in the Competition
- The challenge received 62 system submissions from 17 institutions, with about one-third of submissions coming from industry.Two industry groups withdrew submissions and one group requested anonymisation after automatic evaluation.Source: §3 · p. 3
- The remaining teams identified 20 primary systems from 14 groups for evaluation by automatic metrics and human judges.Each primary system was described in a short technical paper.Source: §3 · p. 3
- Primary systems were compared with a TGEN baseline, spanning a range of architectures and system features.The paper presents an overview of the systems and their main architectural features in Table 3.Source: §3 · p. 3
4 Evaluation Results
Automatic word-overlap metrics favor seq2seq systems, but human evaluation gives a different picture: SHEFF2 leads in naturalness and SLUG in quality. Human results also show broad significance clusters and mixed architecture performance on quality.
4.1 Word-overlap Metrics
- Seq2seq systems lead word-overlap metrics overall, followed by imitation-learning system SHEFF1; template- and rule-based systems mostly rank lowest.TGEN remains very strong: no primary system beats it on all metrics, and SLUG comes closest.Source: §4.1 · pp. 3–4
4.2 Results of Human Evaluation
- Secondary submissions can outperform primary systems and TGEN on word-overlap metrics, although the primary-system comparison identifies TGEN as especially strong.Source: §4.1 · p. 3, §4.2 · p. 4
- SHEFF2 wins naturalness and SLUG wins quality, while the second significance clusters contain 13 and 11 systems, respectively, including TGEN.Table 4 reports TrueSkill measurements for quality and naturalness, with significance clusters and system rank ranges.Source: §4.2 · p. 4, Table 4 · p. 4
- Human evaluation ranks seq2seq systems highest for naturalness, while most other architectures score lower and template-based systems fill the bottom cluster.Source: §4.2 · p. 4
- Human quality results are more mixed by architecture: seq2seq systems with correctness-checking rerankers score high, while those without such mechanisms occupy the bottom two clusters.Source: §4.2 · p. 4
5 Conclusion
The challenge introduced a large dataset and compared diverse NLG systems using automatic and human evaluation. SLUG was the overall winner, while the results showed strengths and trade-offs across system architectures.
5 Conclusion
- The challenge created a dataset an order of magnitude larger than prior publicly available task-oriented NLG datasets and received 62 submissions from 17 institutions.The paper evaluated all entries with five automatic metrics and crowdsourced human evaluation of naturalness and overall quality for 20 primary submissions.Source: §5 · pp. 4–5
- The challenge aimed to assess end-to-end data-driven systems trained on input meaning representations and texts without fine-grained semantic alignments.Source: §5 · p. 4
- SLUG, a seq2seq ensemble with a reranker, is the overall winner, leading in human quality, ranking in the second naturalness cluster, and achieving high automatic scores.SHEFF2 wins naturalness but falls in the last quality cluster; TGEN ranks highest on average in word-overlap metrics and places in the second cluster for both human criteria.Source: §5 · p. 5
- The results show seq2seq systems are capable but require reranking to reach high-quality results; rule-based systems often match or exceed data-driven systems in human evaluation despite weaker automatic metrics.Source: §5 · p. 5
- The authors planned to release system outputs with user ratings and use the data for broader experiments in automatic NLG output-quality estimation.They state that the challenge's large data volume allows a wider range of such experiments than previously possible.Source: §5 · p. 5