PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Evaluation Metrics Used for NLG Systems

Ananya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra

arXiv:2008.12009v2cs.CL

TL;DR

Tracking progress in NLG requires accurate evaluation, but evaluating NLG systems is challenging and multifaceted. This survey categorizes metrics as context-free or context-dependent and concludes that task-specific, context-dependent metrics and robust benchmarks remain needed.

  • Problem

    Accurately quantifying progress in NLG remains difficult because evaluating NLG systems is a multi-faceted challenge.

  • Method

    The survey categorizes evaluation metrics at a high level as context-free and context-dependent metrics.

  • Results

    The survey concludes that current metrics still leave a need for task-specific, context-dependent metrics.

  • Takeaways & Limitations

    The authors recommend building datasets from human judgements, developing interpretable scores, and creating robust benchmarks.

  • Takeaways & Limitations

    Correlations reported for evaluation metrics show high variance, and the use of automatic evaluation metrics has been criticized in many studies.

Abstract

from arXiv · show

The success of Deep Learning has created a surge in interest in a wide a range of Natural Language Generation (NLG) tasks. Deep Learning has not only pushed the state of the art in several existing NLG tasks but has also facilitated researchers to explore various newer NLG tasks such as image captioning. Such rapid progress in NLG has necessitated the development of accurate automatic evaluation metrics that would allow us to track the progress in the field of NLG. However, unlike classification tasks, automatically evaluating NLG systems in itself is a huge challenge. Several works have shown that early heuristic-based metrics such as BLEU, ROUGE are inadequate for capturing the nuances in the different NLG tasks. The expanding number of NLG models and the shortcomings of the current metrics has led to a rapid surge in the number of evaluation metrics proposed since 2014. Moreover, various evaluation metrics have shifted from using pre-determined heuristic-based formulae to trained transformer models. This rapid change in a relatively short time has led to the need for a survey of the existing NLG metrics to help existing and new researchers to quickly come up to speed with the developments that have happened in NLG evaluation in the last few years. Through this survey, we first wish to highlight the challenges and difficulties in automatically evaluating NLG systems. Then, we provide a coherent taxonomy of the evaluation metrics to organize the existing metrics and to better understand the developments in the field. We also describe the different metrics in detail and highlight their key contributions. Later, we discuss the main shortcomings identified in the existing metrics and describe the methodology used to evaluate evaluation metrics. Finally, we discuss our suggestions and recommendations on the next steps forward to improve the automatic evaluation metrics.

1 INTRODUCTION

NLG has expanded rapidly with deep learning, creating a pressing need for reliable evaluation. This survey organizes evaluation metrics, examines their shortcomings, and recommends task-sensitive improvements.

  • Deep learning advanced existing NLG tasks and expanded research into tasks such as image and video captioning.
  • Human evaluation can rank systems, but it requires skilled annotators, elaborate guidelines, and substantial time and expense.
  • Automatic metrics offer faster comparisons, yet BLEU-like metrics often correlate poorly with human judgments and miss task-specific requirements.
  • Metric adoption continued despite criticism, including use on newer tasks for which the original metrics were not designed.
  • Since 2015, at least 36 new metrics have been proposed, compared with about 10 metrics used between 2002 and 2014.
  • The survey highlights evaluation challenges, proposes a metric taxonomy, analyzes shortcomings, and recommends fine-grained task-specific evaluation.

2 VARIOUS NLG TASKS

NLG tasks require metrics that assess more than surface overlap because valid outputs can differ while factuality, relevance, fluency, and task fulfillment vary. The section illustrates these challenges across translation, summarization, question answering, question generation, data-to-text, and dialogue.

  • NLG evaluation is difficult because generated outputs must satisfy multiple criteria across different tasks.
  • Machine Translation: Machine translation metrics must distinguish meaning preservation and grammatical correctness from simple word overlap.
  • Machine Translation: A translation with lower word overlap can have better quality than one closely matching the reference but changing the meaning.
  • Abstractive Summarization: Summarization evaluation must detect factual errors, missing crucial information, and valid alternative phrasing.
  • Question Generation: Question-generation metrics must assess fluency, answerability, specificity, and alignment with the input answer or source.
  • Data-to-Text and Dialogue: Data-to-text and dialogue evaluation additionally require factual coverage, correctness, coherence, fluency, and goal alignment.

3 HUMAN EVALUATION OF NLG SYSTEMS

Human evaluation of NLG systems is multifaceted: evaluators must assess several output qualities while choosing an appropriate setup, scale, context, and evaluator type. These choices affect reliability, cost, speed, and applicability across tasks.

  • Evaluation criteria: NLG evaluation requires simultaneous assessment of a wide set of output qualities rather than a single criterion.The paper frames evaluation as a multifaceted task and discusses both overall task fulfillment and fine-grained quality assessment.
  • Human evaluation setup: Human-evaluation setups vary with budget, annotator availability, speed, and required precision.Possible evaluators include experts, crowdsourced annotators, and end-users, depending on task requirements and evaluation goals.
  • Human evaluation setup: Evaluator choice depends on the task: domain experts are important for specialized content, while end-users can judge dialogue effectiveness.Crowdsourced workers are not preferred when domain knowledge is required, whereas end-users are better suited to real-world effectiveness judgments.
  • Human evaluation setup: Evaluation may use absolute ratings, relative comparisons, rankings, pairwise preferences, binary judgments, or continuous scales.Likert scales commonly use five levels, while continuous scales provide more nuanced judgments and binary scales force clearer decisions.
  • Human evaluation setup: Providing the generated output with its context and references can improve assessment, although fluency can be judged without a reference.A single reference is often insufficient for NLG tasks, making context useful despite increasing evaluator cognitive load.
  • Human evaluation setup: Reliable aggregate scores require multiple evaluators, clear guidelines, and attention to inter-annotator agreement.Agreement can be measured with Cohen’s Kappa, Fleiss Kappa, Krippendorff’s alpha, Jaccard similarity, or F1; disagreement may reflect error, inadequate guidelines, or textual ambiguity.

4 TAXONOMY OF AUTOMATED EVALUATION METRICS

The survey organizes automatic NLG metrics by whether they use context and by whether they are trained or untrained. Context-free metrics are broadly reusable, while context-dependent metrics are usually task-specific and mostly trained.

  • Taxonomy dimensions: The taxonomy first separates automatic metrics into context-free and context-dependent categories.Context-free metrics compare hypotheses with references without using the input context; context-dependent metrics also consider the context.
  • Context-free metrics: Context-free metrics are task-agnostic and easier to adopt because they compare reference and hypothesis word sequences across tasks.Their task independence follows from using the same basic comparison form regardless of the NLG task.
  • Context-dependent metrics: Context-dependent metrics are typically designed for a specific task and may require modifications before adoption elsewhere.For example, a machine-translation metric using a source sentence cannot directly transfer to image captioning or data-to-text generation.
  • Technique-based classification: Metrics are further classified by technique, including trained versus untrained methods and the representations or features they use.Untrained methods may use fixed heuristics over words, characters, or embeddings, whereas trained methods may use engineered features or end-to-end representations.
  • Technique-based classification: Most context-dependent metrics are trained, with only a handful of untrained metrics identified by the survey.The taxonomy applies the trained/untrained distinction to both context-free and context-dependent metrics.

5 CONTEXT-FREE METRICS

Context-free metrics evaluate hypotheses by comparing them with available references, either through fixed heuristic features or task-trained components. Untrained metrics use word-, character-, or embedding-based features, with representative metrics addressing overlap, matching, and semantic similarity.

  • Context-free metrics compare a hypothesis with the available references without considering the input context.
  • They divide into untrained metrics using predefined heuristic features and trained metrics containing learnable components trained for evaluation.
  • Untrained metrics use word-, character-, or embedding-based features.
  • Word-based metrics: BLEU computes clipped n-gram precision, combines precisions across n values, and adds a brevity penalty to discourage short hypotheses.BLEU is corpus-level because its precisions are summed over all hypotheses.
  • Word-based metrics: NIST modifies BLEU by weighting matched n-grams by information gain and changing the combination and brevity-penalty terms.Rare n-grams receive more credit, while common n-grams receive less.
  • Word-based metrics: GTM uses an F-score over word matchings, weighting contiguous matching runs more heavily and approximating maximum match size greedily.The maximum match-size problem is NP-hard for q > 1.
  • Word-based metrics: GTM showed higher correlations with human judgments than BLEU and NIST for machine-translation evaluation.
  • Embedding-based metrics: BERTScore computes token-level precision, recall, and F1 through embedding similarities, and reported competitive human-judgment correlations across 4 NLG tasks.The tasks were machine translation, image captioning, abstractive summarization, and data-to-text generation.

Linear Regression

BEER is a linear-regression metric that combines lexical, syntactic, and word-order features learned from human judgments.

  • BEER uses character n-gram and word-level precision, recall, and F1 features, together with permutation-tree features for word order and fluency.
  • Its unigram features separately represent function words, content words, and all words.
  • BEER applies a simple linear function to these input features.
  • Feature weights are learned by linear regression using human judgments from WMT13.

SVM Regression

BLEND combines existing metrics in an SVM regressor trained against human direct-assessment scores. Its feature set spans lexical, syntactic, and semantic metric categories.

  • BLEND combines existing untrained metrics to improve correlation with human judgments.
  • It uses an SVM regressor with 57 metric scores as features and WMT15 and WMT16 direct-assessment scores as the gold-standard target.
  • The 57 metrics are categorized as lexical, syntactic, and semantic metrics.
  • The lexical category contains 25 metrics representing 9 metric types, including multiple BLEU variants.
  • An ablation study found that combining all categories produced the best results.

Grid search with bagging

Q-Metrics adapt n-gram evaluation for question generation by weighting word categories differently and tuning the resulting metric to maximize correlation with human answerability judgments.

  • Q-Metrics seek better correlation with human judgments on answerability for question generation.
  • They assign different weights to words instead of treating all words equally as standard n-gram metrics do.
  • Question words are divided into function words, Wh-words, named entities, and content words.
  • The metric computes n-gram precision and recall separately for each category, then forms weighted-average P_avg and R_avg values.
  • Weights and δ are tuned through grid search and bagging to maximize correlation with human scores.

Neural networks/ Deep Learning

Neural evaluation metrics combine learned models with lexical, semantic, and sentence-embedding features to assess generated text and captions.

  • Composite metrics: Sharif et al. train a feedforward neural network on combinations of METEOR, CIDEr, WMD, and SPICE features to classify captions as machine- or human-generated.Training uses Flicker30k references as positive samples and captions generated by three models as negative samples.
  • Composite metrics: NNEval extends this feature set by additionally incorporating BLEU(1-4) scores.
  • End-to-end Trained metrics: SIMILE trains a sentence encoder on paraphrase pairs, then computes cosine similarity between reference and candidate embeddings.A length penalty discourages repetitive, overly long generations, with its influence tuned through α.
  • End-to-end Trained metrics: ESIM uses BiLSTM representations, cross-sentence attention, enhanced sentence representations, and pooled hidden states to predict an evaluation score.It is trained on WMT 2016 Direct Assessment human-evaluation data.
  • End-to-end Trained metrics: RUSE regresses a score from InferSent, Quick-Thought, and Universal Sentence Encoder embeddings of the hypothesis and reference.The embeddings provide global sentence information beyond local character- or word-n-gram features.

Transformer based trained metrics

Transformer-based trained metrics encode hypotheses and references jointly or extract multiple semantic signals, then learn scores from human-evaluation data.

  • Transformer-based metrics: Transformer-based metrics replace or complement recurrent architectures, using pretrained models such as BERT, RoBERTa, GPT-2, and XLNet.
  • BERT for MTE: BERT for MTE concatenates the reference and hypothesis, uses the [CLS] representation, and applies an MLP regressor whose encoder is jointly fine-tuned.
  • BERT for MTE: Pair-encoding the candidate and reference together improves correlations with human judgments over RUSE, according to the authors.
  • BLEURT: BLEURT pretrains BERT on synthetically perturbed sentence pairs using lexical, back-translation, entailment, and perturbation signals before supervised regression fine-tuning.It achieves state-of-the-art performance on WMT and WebNLG after fine-tuning on those datasets.
  • BLEURT: BLEURT achieves state-of-the-art performance on WMT and WebNLG challenges after fine-tuning on those datasets.
  • NUBIA: NUBIA matches or outperforms existing metrics on machine translation and image-captioning correlations with human judgments.

6 CONTEXT-DEPENDENT METRICS

Context-dependent metrics incorporate task-specific input context alongside hypotheses, using word-based or embedding-based features and learned discriminative models.

  • 6 CONTEXT-DEPENDENT METRICS: Context-dependent metrics are specific to their corresponding tasks because input context varies, unlike context-free metrics adopted across many tasks.
  • Untrained metrics: Untrained context-dependent metrics use either word-based or embedding-based features from the hypothesis and its context.
  • Word based metrics: ROUGE-C compares a candidate summary with the document, making it useful when a reference summary is unavailable.
  • Word based metrics: For query-focused summarization, ROUGE-C combines document-context and query-focused information using a tunable weighting factor.
  • Word based metrics: PARENT matches hypothesis n-grams against both references and table records, weighting table matches by entailment probabilities before computing an F-score.
  • End-to-end Trained metrics: Learned context-free metrics use neural architectures ranging from MLPs to transformers, with many applications in dialogue evaluation and image captioning.
  • LEIC: LEIC evaluates captions discriminatively by incorporating the image, unlike overlap-based metrics that compare only candidate and reference captions.It combines image and text representations and trains a classifier on human and machine-generated captions, including augmented negative examples.

Transformer based metrics

Transformer-based dialogue metrics use contextual embeddings and classifiers to score responses, including reference-free evaluation and harder negative examples.

  • RUBER with BERT embeddings: RUBER with BERT embeddings replaces RUBER’s word2vec representations with contextualized BERT embeddings.
  • RUBER with BERT embeddings: The unreferenced model uses pooled contextual embeddings and a binary MLP classifier trained with cross-entropy, without fine-tuning BERT.
  • MaUde: MaUde combines the transformed context representation with the response representation and feeds them to a sigmoid classifier.
  • RoBERTa-eval: RoBERTa-eval encodes dialogue context and response jointly, then uses an MLP classifier to produce a score from 1 to 5.It is trained on DailyDialog human response-quality annotations with randomly sampled and model-generated responses.

7 STUDIES CRITICISING THE USE OF AUTOMATIC EVALUATION METRICS

Studies criticize automatic NLG metrics for weak and unstable alignment with human judgements, limited interpretability, task-specific biases, and poor coverage of linguistic nuances.

  • Poor correlations: Automatic metrics often correlate poorly with human judgements across NLG tasks, with early studies finding negative fluency correlations and moderate-to-low adequacy correlations.These findings have been reported for BLEU, NIST, SSA, GTM, and LSA, among other metrics.
  • Poor correlations: Correlation estimates vary substantially across studies because human-rating procedures and metric-specific parameter settings are not always consistently reported.Standardization efforts such as the WMT shared task reduce some external variation but do not eliminate reliability issues.
  • Poor correlations: Correlations become unreliable when ranking a small set of systems with similar capability, including the 24 metrics examined in WMT-19.This problem can occur even when evaluation datasets and human-rating setups are standardized.
  • Uninterpretability of scores: Automatic metrics assign a single score that often cannot reveal whether an output is fluent, adequate, coherent, relevant, or informative.BLEU can assign identical scores to different n-gram permutations, including incorrect variants, and can penalize valid translations that differ from references.
  • Bias in the metrics: Metrics exhibit biases and limited generalization, including BLEU’s preference for n-gram-based systems, GAN evaluators’ poor performance on unseen systems, and ADEM’s narrow score range.ADEM’s response encodings also show high conicity, with little separation between relevant and irrelevant responses.
  • Poor adaptability across Tasks: Metrics often fail to adapt across tasks or capture task nuances: reference-overlap metrics are unsuitable for dialogue, while other evaluations miss factual consistency, faithfulness, or response diversity.A correct dialogue response may have no n-gram overlap with the reference, and task-specific metrics can still omit important criteria.

8 EVALUATING EVALUATION METRICS

The survey evaluates evaluation metrics primarily by comparing their scores with human judgements and explains how correlation measures behave under different data and ranking conditions.

  • Methodology: The most widely used evaluation approach computes correlations between metric scores and human judgements on generated or synthetically constructed outputs.Human judgements may reflect task-specific criteria, and the outputs can be system-generated or synthetically crafted.
  • Correlation measures: Rank-based correlations can unfairly penalize metrics when system scores differ only slightly, motivating WMT’s shift to system-level Pearson correlation since 2014.WMT also uses a custom Kendall’s τ variant that discards translation pairs with similar human scores before counting concordant and discordant pairs.
  • Correlation measures: Pearson can miss nonlinear relationships and is highly sensitive to outliers, while Spearman and Kendall’s τ can also be sensitive to outliers in the middle of the distribution.For perfect monotonic but nonlinear relationships, rank correlations can be perfect while Pearson is lower.
  • Statistical significance: Correlation magnitude alone does not establish statistical significance: a coefficient of 0.7 from 500 samples is more significant than the same coefficient from 5 samples.P-values test evidence against a zero population correlation and depend on sample size and data variation.
  • Other evaluation considerations: Evaluation outcomes can vary with reference quality and judge subjectivity, while automatic and human evaluations may produce different system rankings.Studies report that more good-quality references lead to better correlations and that fluency is often more reliably determined than other qualities.

9 RECOMMENDATIONS (POSSIBLE FUTURE RESEARCH DIRECTIONS)

The survey recommends shared datasets and code, context-dependent and interpretable metrics, and robust benchmarks to address weaknesses in current NLG evaluation.

  • Developing a common code base for evaluation metrics: A common code base would make metric evaluation more reproducible and allow fairer comparisons across implementations and systems.It would also make a wider range of metrics easier to use rather than favoring metrics whose code is already available.
  • Building datasets containing human judgements: Researchers should build datasets pairing contexts, references, proposed responses, and human scores across multiple criteria, tasks, and languages.The survey notes a continuing scarcity of such datasets and identifies standardized shared tasks as a useful model.
  • Developing task-specific context-dependent metrics: The survey recommends task-specific, context-dependent metrics because context is needed to assess coherence, factual consistency, relevance, and correctness.Reference or semantic overlap alone is inadequate for dialogue, summarization, question answering, and data-to-text generation.
  • Developing more interpretable metrics: More interpretable metrics should provide separate scores for criteria such as fluency, adequacy, coherence, and informativeness instead of one overall score.Fine-grained scores could provide clearer directions for system improvement and help users compare systems with different strengths.
  • Creating robust benchmarks for evaluating evaluation metrics: The field should create adversarial benchmarks that test whether metrics resist high-overlap but irrelevant or factually incorrect outputs and reveal metric biases.Recommended analyses include adversarially crafted responses, white-box attacks, and checks for training-data and pretrained-data biases.
  • Creating robust benchmarks for evaluating evaluation metrics: Researchers should also scrutinize correlation measures because recent studies show that these measures can be unreliable in certain situations.This recommendation complements broader efforts to test metric robustness and comparability.

10 CONCLUSION

The survey organizes rapidly expanding NLG evaluation research into a taxonomy, reviews evaluation practices and shortcomings, and recommends directions for more robust, interpretable, task-sensitive metrics.

  • 10 CONCLUSION: The need for improved metrics remains because NLG spans multiple tasks and criteria, while automatic evaluation must quantify progress accurately.The survey frames automatic evaluation as challenging because generated outputs require assessment across a wide variety of criteria.
  • 10 CONCLUSION: The survey categorizes metrics as context-free or context-dependent, then distinguishes trained and untrained approaches using word-, character-, or embedding-based information.This taxonomy organizes existing metrics by context sensitivity, training status, and representational features.
  • 10 CONCLUSION: The survey reviews Pearson, Spearman’s, and Kendall’s Tau correlations and significance tests, emphasizing that evaluation measures can be unreliable in some situations.It gives Pearson correlation as appropriate mainly when a linear relationship between variables is expected.
  • 10 CONCLUSION: Existing metrics are criticized for poor human-judgment correlation, uninterpretability, inherent biases, limited cross-task adaptability, and weak capture of task-specific nuances.The survey also notes that many current metrics are context-free even though context is important in numerous NLG tasks.
  • 10 CONCLUSION: The authors recommend a common code base, task-specific datasets with fine-grained human judgments, context-dependent metrics, interpretable scores, and robust benchmarks.These recommendations target reproducibility, precise improvement directions, and critical examination of metric shortcomings and biases.
Loading 2008.12009v2…