Source-linked AI summary
Why We Need New Evaluation Metrics for NLG
Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, Verena Rieser
TL;DR
Automatic NLG metrics are widely used despite limited evidence that they reflect human preferences. This paper compares word-based, semantic, and grammar-based metrics across recent end-to-end data-driven NLG settings, finding weak sentence-level correspondence but more reliable system-level results and data- and system-specific performance.
Problem
Automatic metrics are cheaper and faster than human evaluation, but their correlation with human preferences in NLG is often insufficient.
Method
The paper evaluates 21 automatic metrics, including grammar-based measures, across three systems, three datasets, and two domains using human ratings for comparison.
Results
Automatic metrics do not sufficiently reflect human ratings, perform differently by data and system, and are particularly weak at distinguishing medium- and good-quality outputs.
Takeaways & Limitations
Metrics can support error analysis and system development by identifying poorly performing cases and providing reliable system-level results.
Takeaways & Limitations
The evaluated datasets are fairly small, containing between 404 and 2.3k human references per domain.
Abstract
from arXiv · showhide
The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.
1 Introduction
The paper examines whether automatic NLG metrics reflect human preferences, focusing on recent end-to-end, data-driven systems and broad comparisons across metrics, systems, datasets, and domains.
- Motivation: Automatic metrics are cheaper and faster than human evaluation, but their use is sensible only if they sufficiently correlate with human preferences.They are also used for automatic benchmarking and algorithm tuning.
- Scope: The study targets end-to-end data-driven NLG systems that generate word-by-word and learn from noisy data, allowing ungrammatical output.These systems jointly learn sentence planning and surface realisation from non-aligned data.
- Study design: The evaluation compares 21 automated metrics, including novel grammar-based metrics, across three systems, three datasets, and two domains.The study also includes detailed error analysis and publicly releases associated code and data.
- Motivation: The study aims to establish the need for new automatic metrics that are independent of systems and data.The paper presents the evaluation as a comprehensive study of recent data-driven NLG.
3 Datasets
The study uses crowdsourced datasets for spoken-dialogue utterance generation, covering San Francisco hotels and restaurants and Cambridge restaurants.
- Dataset structure: Each data instance contains one meaning representation and one or more human-produced natural-language references.The BAGEL example maps an inform MR to a moderately priced restaurant utterance.
- Preprocessing: The study uses lexicalised SFHOTEL and SFREST data and a partially lexicalised BAGEL dataset, replacing some names with placeholders.Proper names and place names are replaced by “X” in the relevant outputs.
- SFHOTEL and SFREST: SFHOTEL and SFREST cover San Francisco hotels and restaurants with eight dialogue-act types and twelve attributes per domain.The datasets include both shared and domain-specific attributes.
- SFHOTEL and SFREST: SFHOTEL contains 2.3K unique human utterances and 875 unique system outputs, while SFREST contains 1.6K unique human utterances and 1,181 unique system outputs.Around 5K human references were collected for each domain.
- BAGEL: BAGEL contains 202 aligned MR-reference pairs with two corresponding references each and covers a subset of SFREST’s restaurant domain.Its domain includes only the inform act and eight attributes.
4 Metrics
The paper evaluates reference-based word metrics, semantic similarity, and reference-free grammar-based measures targeting readability and grammaticality.
- Word-based metrics: Word-based metrics compare system outputs with human references, generally treating higher scores as better or more similar to those references.The set includes TER, BLEU, ROUGE, NIST, LEPOR, CIDEr, and METEOR.
- Word-based metrics: The evaluated word metrics range from n-gram overlap to measures incorporating term-frequency weighting and semantic similarity.TER uses a reversed scale relative to the other described metrics.
- Semantic similarity: Semantic Text Similarity uses distributional similarity and Latent Semantic Analysis, complemented by semantic relations extracted from WordNet.The measure is included alongside word-overlap metrics.
- Grammar-based metrics: Grammar-based metrics do not rely on ground-truth references, and this paper is the first to apply them to sentence-level NLG evaluation.The study focuses on readability and grammaticality.
- Grammar-based metrics: Readability is measured with Flesch Reading Ease and related length measures, while grammaticality is approximated using misspellings and Stanford parser scores.Higher Reading Ease indicates easier reading; lower misspelling counts indicate greater grammatical correctness.
5 Human Data Collection
Human rankings were collected by having English-speaking crowdworkers compare system-generated utterances alongside their meaning representations and rate multiple quality dimensions.
- Rating procedure: Crowdworkers rated paired system utterances on a 6-point Likert scale for informativeness, naturalness, and overall quality.The meaning representation was presented with the two generated utterances.
- Rater controls: Each system output was scored by three crowdworkers, with randomized utterance order and a maximum of 20 evaluations per worker.Workers were selected from English-speaking countries and confirmed English as their native language.
- Rating reliability: The overall intra-class correlation coefficient across the three datasets was 0.45 (p < 0.001), indicating moderate agreement.Agreement differed consistently by system and dataset, with lower agreement for LOLS than for RNNLG and TGEN.
- Rating reliability: Agreement was highest for SFHOTEL, followed by SFREST and BAGEL.This comparison concerns inter-annotator agreement across the three datasets.
6 System Evaluation
System-level results suggest relationships between automatic metrics and human ratings, but these relationships vary across systems and datasets. Aggregate scores can therefore obscure poor individual outputs.
- System performance is dataset-specific, with LOLS outperforming TGEN on BAGEL but RNNLG outperforming LOLS on SFREST and SFHOTEL for WBMs.
- Human informativeness ratings follow the same system-level pattern as WBMs, while SIM appears related to human quality ratings.
- GBMs appear related to naturalness and quality: simpler utterances receive higher naturalness, whereas longer or more complex utterances receive lower quality ratings.
- More grammatical utterances are on average evaluated as more natural by humans.
- Average system scores may conceal worst-case outputs, motivating utterance-level comparisons for each MR-system output pair.
7 Relation of Human and Automatic Metrics
At utterance level, automatic metrics generally correlate weakly with human ratings and their performance depends on the dataset, system, and rating conditions. Relative-ranking accuracy is often near random, although quantising metric scores can improve results in some settings.
- The analysis uses Spearman correlations on dataset-system splits, median scores from three raters, and Williams’ tests for comparing correlations.
- No metric achieves even a moderate correlation with human ratings independently of dataset, system, or rating aspect.
- The highest positive correlation is wps with informativeness for TGEN on BAGEL (ρ = 0.33, p < 0.01), but wps reaches ρ ≤.18 on other datasets.
- A random score has a highest correlation of ρ = 0.09, providing a sanity check for the correlation analysis.
- Most metrics are not significantly more accurate than random for relative human ratings, with metrics reaching 30.6–49.8% accuracy versus random scores of 25.4–44.5%.
- Relative-ranking performance is dataset-specific: metrics perform best on BAGEL, show mixed performance on SFHOTEL, and perform worst on SFREST.
- Quantising metric scores to the human-rating scale makes all WBMs significantly better than random for predicting informativeness on SFREST.
8 Error Analysis
The error analysis finds that automatic metrics distinguish clearly poor outputs better than average or good ones, while correlations vary with data characteristics and metric design.
- 8.1 Scales: Metrics correlate significantly better with low-rated informativeness and naturalness than with average or good ratings.Low-informativeness utterances reach 0.3 ≤ρ ≤0.5, whereas average and high informativeness barely reach ρ ≤0.2.
- 8.1 Scales: 79% of outputs are rated good for informativeness, 64% for naturalness, and 58% for quality, while low ratings never exceed 7%.This imbalance could explain low overall correlations and the RNNLG system’s poor correlation.
- 8.2 Impact of Target Data: BAGEL differs from SFREST and SFHOTEL in reference length, word complexity, and meaning-representation types, affecting how metrics reflect human ratings.BAGEL contains only inform meaning representations, whose utterances are longer and correlate more with informativeness and naturalness ratings.
- 8.2 Impact of Target Data: Metrics perform most reliably on BAGEL despite its greater generation complexity, although the correlation remains weak.BAGEL has two human references per meaning representation, compared with 5.35 references per meaning representation on average for SFHOTEL and SFREST.
- 8.3 Example-based Analysis: Word-based metrics agree moderately with humans on poor-quality output but fail to distinguish medium- and good-quality output.Examples show reference-overlap metrics missing semantically appropriate outputs, while grammar-based metrics better capture some readable, grammatical cases.
9 Related Work
Related studies report correlations ranging from weak to strong, with metrics generally more reliable at system level and better at capturing surface realization than sentence planning.
- 9 Related Work: Previous studies mainly evaluated word-based metrics, whereas this paper is the first described here to consider grammar-based metrics.Earlier work separately targeted surface realization or sentence planning dimensions.
- 9 Related Work: Reported correlations range from weak to strong, but prior studies often omit significance tests comparing metric correlations.The strongest metrics generally combine word overlap, semantic similarity, and term-frequency weighting.
- 9 Related Work: Metrics tend to be reliable at system level, less reliable at sentence level, and better at capturing realization than sentence planning.This pattern is consistent across the related results summarized in the section.
10 Conclusions
The conclusions show that current automatic metrics insufficiently reflect human ratings and that their performance depends on the data and system. They remain useful for error analysis and system development at system level.
- 10 Conclusions: State-of-the-art automatic metrics do not sufficiently reflect human ratings of NLG outputs, strengthening the case for human evaluation.This finding contrasts with the documented trend toward relying on automatic evaluation.
- 10 Conclusions: Automatic metrics are particularly weak at distinguishing medium- and good-quality outputs, and their performance is data- and system-specific.The error analysis partly attributes this weakness to human judgements and metrics using different scales.
- 10 Conclusions: Automatic metrics can help error analysis by finding cases where a system performs poorly.This use is supported even though the metrics do not reliably mirror human ratings in general.
- 10 Conclusions: Reliable system-level results suggest that automatic metrics can support system development.
11 Future Directions
The paper motivates more advanced, reference-less evaluation while identifying limitations in reference assumptions, cross-domain generalization, and dataset size.
- 11 Future Directions: Word-based metrics assume human references are correct and complete, while grammar-based metrics avoid reference quality but can reward irrelevant grammatical output.An ensemble combining both metric families achieved high within-domain correlation but insufficient cross-domain performance.
- 11 Future Directions: Reference-less evaluation directions include dialogue-context quality, task success, discriminative models, and quality prediction from meaning representations.The paper presents grammar-based metrics as an initial step toward reference-less NLG evaluation.
- 11 Future Directions: The evaluated datasets are fairly small, containing between 404 and 2.3k human references per domain.The paper notes ongoing efforts to release larger and more diverse datasets.
Appendix A: Detailed Results
The appendix reports detailed system performance, metric–human-rating correlations, significance conventions, and analyses by dataset, system, utterance quality, and MR type.
- Evaluation details: Human-rating reliability across the three datasets is measured with ICC, while system performance is summarized using means, standard deviations, and significance tests.The ICC assesses inter-observer reliability; system comparisons mark statistically significant differences at p < 0.05.
- Reporting conventions: Bold font denotes a significantly stronger correlation when comparing two systems on the same dataset.This convention applies to the appendix’s comparative correlation results.
- Correlation analyses: Spearman-correlation tables report metric associations with human ratings separately for datasets, systems, and individual dataset–system combinations.Statistical significance is marked with an asterisk at p < 0.05.
- Relative-rating prediction: Relative-rating accuracy is reported separately from correlation, with asterisks identifying statistically significant results.This table evaluates whether metrics predict relative human ratings.
- Subgroup analyses: Additional analyses compare metric correlations for low-quality utterances against good and average utterances, and for inform versus non-inform MRs.Bold formatting identifies significantly stronger correlations in the designated comparison group.