Why We Need New Evaluation Metrics for NLG

1 Introduction

• Up to 60% of NLG research published between 2012–2015 relied on automatic metrics, which are cheaper and faster than human evaluation but need to correlate with human preferences.Prior studies in NLG and related fields have often found that such metrics do not correlate sufficiently with human judgements (Liu et al., 2016; Kilickaya et al., 2017).Source: §1 · p. 1
• The study compares 21 automatic metrics, including novel grammar-based metrics, across three end-to-end approaches, two domains, and three datasets, finding metrics unreliable at sentence level but useful at system level.The authors also make the associated code, data, and detailed analysis results publicly available.Source: §1 · p. 1
• The compared systems can produce ungrammatical output because they generate word-by-word and learn from noisy data.Source: §1 · p. 1

2 End-to-End NLG Systems

• The study targets end-to-end, data-driven NLG that jointly learns sentence planning and surface realisation from non-aligned data, avoiding costly alignment between meaning representations and references.Parallel datasets collected through crowdsourcing can support rapid development of NLG components in new domains (Novikova et al., 2016).Source: §2 · pp. 1–2
• RNNLG uses an MR-conditioned LSTM for joint sentence planning and surface realisation, while TGEN incrementally generates dependency trees and uses a separate rule-based realiser (Wen et al., 2015; Dušek and Jurčíček, 2015).Source: §2 · p. 2
• LOLS learns sentence planning and surface realisation through imitation learning, using BLEU and ROUGE as non-decomposable loss functions.Source: §2 · p. 2

3 Datasets

• Dataset instances pair an MR with one or more human-written references, such as “X is a moderately priced restaurant in X” for inform(name=X, area=X, pricerange=moderate, type=restaurant).SFHOTEL and SFREST use lexicalised versions, while BAGEL is partially lexicalised, with proper and place names replaced by “X,” corresponding to generated outputs.Source: §3 · p. 2
• SFHOTEL and SFREST cover San Francisco hotels and restaurants, with about 5K human references per domain, 2.3K unique SFHOTEL utterances, 1.6K unique SFREST utterances, and 875 and 1181 unique system outputs, respectively.Both domains have eight dialogue-act types and 12 attributes, some shared and some domain-specific (including food and kids-allowed for restaurants, and hasinternet and dogs-allowed for hotels).Source: §3 · p. 2
• BAGEL covers Cambridge restaurants with 202 aligned MR-reference pairs and two corresponding references per MR; its domain includes only the inform act and eight attributes.BAGEL is a subset of SFREST.Source: §3 · p. 2

4.1 Word-based Metrics (WBMs)

• WBMs compare system outputs with human references using word-overlap metrics and semantic similarity.The listed metrics include TER, BLEU, ROUGE, NIST, LEPOR, CIDEr, METEOR, and SIM; TER’s scale is reversed, and SIM uses distributional similarity, LSA, and WordNet relations.Source: §4.1 · pp. 2–3

4.2 Grammar-based metrics (GBMs)

• The authors introduce GBMs for sentence-level NLG evaluation, using readability and grammaticality measures that do not rely on ground-truth references.They describe this as the first consideration of GBMs for sentence-level NLG evaluation.Source: §4.2 · p. 3
• Readability is measured with Flesch Reading Ease and text-length and syllable measures, with higher RE indicating easier-to-read utterances.Other listed measures capture complexity, including characters per utterance or word, words per sentence, syllables per sentence or word, and polysyllabic words per utterance or word.Source: §4.2 · p. 3
• Grammaticality is approximated by misspelling counts and Stanford parser scores, with fewer misspellings and higher parser scores indicating more grammatical output.The authors note that the parser score is not designed to measure grammaticality and propose using purpose-built grammar-scoring functions when available.Source: §4.2 · p. 3

5 Human Data Collection

• Crowdworkers saw a meaning representation alongside two utterances from different systems and scored each on a 6-point Likert scale for informativeness, naturalness, and overall quality.Quality was defined in terms of grammatical correctness and fluency.Source: §5 · p. 3
• Three crowdworkers scored each system output; utterance order was randomized, workers were limited to 20 utterances, and only native English speakers in English-speaking countries were selected.Country selection was based on IP addresses, and workers confirmed that English was their native language.Source: §5 · p. 3
• Overall ICC was 0.45 (p < 0.001), indicating moderate agreement; agreement was lower for LOLS than for RNNLG and TGEN, and highest for SFHOTEL, followed by SFREST and BAGEL.The dataset details are provided in supplementary material.Source: §5 · p. 3

6 System Evaluation

• System performance is dataset-specific: LOLS beats TGEN on BAGEL, but RNNLG beats LOLS on SFREST and SFHOTEL in WBMs.Human informativeness ratings follow the same pattern as WBMs.Source: §6 · pp. 3–4
• WOMs produce similar results except SIM in BAGEL and SFREST; SIM appears related to human quality ratings.Most GBMs show similar trends at varying significance, while RE shows inverse results.Source: §6 · pp. 3–4
• GBMs relate to naturalness and quality: simpler utterances and greater grammaticality associate with naturalness, while greater complexity associates with lower quality.Readability and word length indicate simplicity; length, word, syllable, and polysyllable measures indicate complexity.Source: §6 · p. 4
• Average system-level scores can conceal worst-case outputs, motivating correlation analysis for each MR–system output pair at utterance level.The initial results suggest a relationship between automatic metrics and human ratings at system level.Source: §6 · p. 4

7.1 Human Correlation Analysis

• No metric achieves even moderate correlation with human ratings across datasets, systems, or rating aspects, and correlation results are system- and dataset-specific.The analysis uses Spearman correlation, splitting data by dataset and system and taking the median of three human ratings.Source: §7.1 · p. 4
• Word-based metrics correlate better with informativeness, while grammar-based metrics correlate better with quality and naturalness; human quality and naturalness ratings correlate at ρ = 0.81.Word-based metrics produce similar results, whereas grammar-based metrics show greater diversity.Source: §7.1 · p. 4
• For TGEN on BAGEL, word count (wps) correlates with informativeness at ρ = 0.33 (p < 0.01), while its correlation on other datasets is weak (ρ ≤ .18).For LOLS, the wps correlation with informativeness ratings is insignificant. A random score has close-to-zero correlation with human ratings (highest ρ = 0.09).Source: §7.1 · p. 5

7.2 Accuracy of Relative Rankings

• Metrics achieve 30.6–49.8% relative-ranking accuracy versus 25.4–44.5% for a random score, with most metrics not significantly better and performance varying by dataset.Metrics perform best on BAGEL, show mixed performance on SFHOTEL, and perform worst on SFREST.Source: §7.2 · pp. 5–6
• After quantizing metric scores to the human-rating scale, all word-based metrics significantly outperform the random baseline for predicting informativeness on SFREST.The task compares two outputs for the same meaning representation against median human ratings and allows ties, accommodating equally rated outputs.Source: §7.2 · pp. 5–6

8.1 Scales

• 79% of outputs were rated good for informativeness, 64% for naturalness, and 58% for quality, while low ratings did not exceed 7% in total, helping explain weak overall correlations and RNNLG’s poor correlation.RNNLG contains very few instances of low user ratings.Source: §8.1 · p. 6

8.3 Example-based Analysis

• WOMs score a grammatically correct, meaning-preserving LOLS output rated median 6 by humans below 1.5, while SIM scores it 5.WOMs penalize the output’s low overlap with references; SIM measures distributional semantic similarity between the MR and output.Source: Table 5 · p. 7, §8.3 · p. 8
• Examples 2 and 3 receive low scores from metrics and humans, while Example 4 is rated highly by humans and has SIM=4 despite WOM failure; its GBM readability mean is 4 and parsing score 3.5.Humans are sensitive to ungrammatical output and missing information, and Example 2 is described as worse than Example 3 despite both receiving median human ratings of 1.Source: Table 5 · p. 7, §8.3 · p. 8

8.1 Scales

• Utterances rated low in informativeness and naturalness correlate significantly better (p < 0.05) with automatic metrics than average- or good-rated ones; low-informativeness correlations are 0.3 ≤ρ ≤0.5 versus at most ρ ≤0.2.The passage also reports the same correlation pattern for quality and naturalness ratings.Source: §8.1 · p. 6

8.2 Impact of Target Data

• BAGEL is more complex, with shorter, more polysyllabic references and only inform MRs, yet metrics perform most reliably on it, although correlation remains weak.BAGEL has two human references per MR versus 5.35 on average for SFHOTEL and SFREST; more references may inflate word-based metric scores. Crowdsourced grammatical errors can mislead corpus metrics, while the parsing score correlates with human ratings at |ρ| = .31.Source: §8.2 · p. 7

• Earlier studies mainly evaluated word-based metrics; this study was the first to consider grammar-based metrics.Source: §9 · p. 8
• Reported correlations ranged from weak to strong, with metrics more reliable at system level than sentence level.Source: §9 · p. 8
• Metrics captured realization better than sentence planning, and stronger performers tended to combine word overlap, semantic similarity, and term-frequency weighting.Source: §9 · p. 8
• Most prior studies did not report whether differences between metric correlations were statistically significant.Source: §9 · p. 8

• Automatic NLG metrics do not sufficiently reflect human ratings, underscoring the need for human evaluation contrary to the trend identified by Gkatzia and Mahamood (2015).Source: §10 · p. 8
• Metrics are particularly weak at distinguishing medium- from good-quality outputs, partly because human judgements and metrics use different scales.Source: §10 · p. 8
• Metric performance is data- and system-specific.Source: §10 · p. 8
• Automatic metrics can help identify poorly performing cases in error analysis, and their reliable system-level results can support system development.Source: §10 · pp. 8–9

11 Future Directions

• Word-based metrics assume references are correct and complete, while grammar-based metrics avoid reference quality but can reward fluent output unrelated to input.A WBMs-and-GBMs ensemble correlated highly with human judgments within one domain, but its cross-domain performance was insufficient.Source: §11 · p. 9
• Future metrics should assess dialogue-context quality (Duˇsek and Jurˇc´ıˇcek, 2016), task-success impact (Rieser et al., 2014; Gkatzia et al., 2016; Hastie et al., 2016), discriminative outputs (Kannan and Vinyals, 2017), or reference-less quality.Grammar-based metrics are presented as a first step toward reference-less NLG evaluation; current work predicts quality from the source meaning representation alone.Source: §11 · p. 9
• Datasets were small (404–2.3k human references per domain), and de-lexicalised training risks ungrammatical lexicalisation and train-test overlap; larger, more diverse datasets are being pursued.The cited risks are associated with de-lexicalised training (Wen et al., 2015; Sharma et al., 2016); larger-data efforts include Novikova et al. (2016, 2017).Source: §11 · p. 9

• Table 7 reports human-rating ICC across three datasets, while Table 8 reports system performance across all datasets.Source: Table 7 · p. 13, Table 8 · p. 13
• Tables 9–11 report Spearman correlations between metrics and human ratings by individual dataset and system, by dataset, and by system.Source: Table 9 · p. 14, Table 10 · p. 15, Table 11 · p. 15
• Table 12 reports metric accuracy in predicting relative human ratings.Source: Table 12 · p. 16
• Table 13 compares WBM–human-rating correlations for Bad-bin utterances versus Good and Average bins, and Table 14 compares correlations for inform versus other MR types.Source: Table 13 · p. 17, Table 14 · p. 17