A Survey of Evaluation Metrics Used for NLG Systems

1 INTRODUCTION

• Deep Learning advanced existing NLG tasks and enabled newer applications including image and video captioning, contributing to a wider and more diverse NLG landscape.The introduction identifies datasets and benchmarks, improved Deep Learning, and cheaper cloud compute as factors behind broader NLP and NLG progress.Source: §1 · p. 2
• Human evaluation can rank NLG systems but requires skilled annotators, elaborate guidelines, and substantial time and expense, creating a bottleneck for rapid model development.Researchers instead use automatic metrics such as BLEU to obtain quick, easy comparisons between systems.Source: §1 · pp. 2–3
• Automatic metrics such as BLEU remained widely adopted despite evidence that they correlate poorly with human judgments, as Figure 1 illustrates through continued citation growth after major criticisms.Figure 1 plots yearly citations for popular metrics and marks the publication years of major criticisms, which did not reduce adoption.Source: §1 · p. 3, Figure 1 · p. 3
• Existing metrics are often unsuitable for newer tasks because they miss task-specific requirements, such as answerability and faithfulness in question generation or goal fulfillment in dialogue.BLEU was not designed to check these properties, and fluency alone is insufficient for goal-oriented dialogue.Source: §1 · p. 3
• Since 2015, at least 36 new automatic NLG evaluation metrics have been proposed, compared with about 10 metrics in use from 2002 to 2014.The survey attributes this surge to broader NLG tasks, infeasible large-scale human evaluation, and inadequate existing metrics.Source: §1 · p. 4

1.1 Goals of this survey

• The survey aims to highlight NLG evaluation challenges and create a taxonomy of existing metrics, emphasizing that evaluation criteria vary across tasks and preclude one universally reusable metric.Distinguishing correct from incorrect outputs requires knowledge of language, domain, and task; summarization additionally involves informativeness, non-redundancy, coherence, and structure.Source: §1.1 · p. 4
• The survey proposes a taxonomy based on metric methodologies to organize existing metrics and clarify developments in the field.The taxonomy distinguishes metrics by context use, supervision, training requirements, and features.Source: §1.1 · p. 4
• Existing NLG metrics often correlate poorly with human judgments, remain uninterpretable, exhibit biases, and miss language nuances.Source: §1.1 · p. 5
• The survey reviews how evaluation metrics are assessed, including correlation measures, statistical hypothesis tests, and tests of adversarial robustness.Source: §1.1 · p. 5
• The survey recommends fine-grained, task-specific evaluation, self-explainable metrics, and datasets containing human judgments across evaluation criteria.Source: §1.1 · p. 5

Overview

• Across NLG tasks, evaluation must assess multiple criteria because generated outputs are required to satisfy a wide variety of requirements.The survey uses informal examples to build intuition for these evaluation challenges before formally listing human evaluation criteria.Source: §2 · p. 5

Machine Translation

• Machine Translation evaluation must recognize meaning-preserving alternatives, penalize semantic changes, and assess fluency despite few available reference translations.Higher word overlap can correspond to worse translation quality, and some substitutions are contextually appropriate while others alter meaning.Source: §2 · p. 5

Abstractive Summarization

• Abstractive Summarization evaluation must distinguish factual errors, missing crucial information, and valid summaries that differ substantially in wording.Unlike translation, summaries may omit source information, but they must remain coherent, factual, and focused on important content.Source: §2 · p. 7

Free-form Question Answering

• Free-form Question Answering evaluation must check fluency, question relevance, factual correctness, and consistency with supplied background information.An answer may address the question but contradict the document, or be correct without addressing the specific question.Source: §2 · p. 7

Question Generation

• Question Generation evaluation must assess grammatical fluency, answerability from the source, and specificity to the answer, with multimodal settings adding further difficulty.Knowledge-based generation may also require domain or common-sense knowledge to recognize valid relations such as “tags” corresponding to “short descriptions.”Source: §2 · p. 7

Data to Text Generation

• Data-to-Text evaluation must judge fluency, adequate verbalization, factual correctness, complete coverage of relevant fields, and acceptance of alternative phrasing.A fluent description can be correct and complete even when it differs from the reference wording.Source: §2 · p. 8

Dialogue Generation

• Dialogue Generation evaluation must allow multiple diverse responses while judging fluency, contextual coherence, engagement, goal alignment, and persona consistency.Word overlap with context does not ensure coherence, and repetition or vague responses can make otherwise valid conversations monotonous.Source: §2 · p. 8

Image Captioning

• Image Captioning evaluation must verify fluency, important entities, and accurate relations without requiring exhaustive image coverage or favoring longer captions.Concise and descriptive captions can both be appropriate when they accurately represent important image content.Source: §2 · p. 8

• Image captions should accurately describe images without unnecessary details and remain fair to shorter, concise captions.Source: §2 · p. 8
• Other NLG tasks include spelling and grammar correction, paraphrase generation, video captioning, text simplification, code generation, and humour generation, but the survey excludes them from detailed discussion because fewer evaluation metrics target them.Source: §2 · p. 8

3 HUMAN EVALUATION OF NLG SYSTEMS

• Human evaluation is multi-faceted because NLG outputs must be assessed across a wide set of qualities.The section first describes human-evaluation procedures and then defines desired output qualities for different NLG tasks.Source: §3 · p. 9

3.1 Human Evaluation Setup

• Human-evaluation setups vary by resources and goals, selecting evaluators, rating scales, context and references, comparisons, rationales, aggregation, and agreement checks.Experts, crowdsourced annotators, or end-users may be used; Likert, continuous, binary, post-editing, absolute, relative, and pairwise formats are described. Context often remains necessary because references may not capture all relevant information, while clear guidelines and quality checks support inter-annotator agreement.Source: §3.1 · p. 9

3.2 Criteria used for Evaluating NLG systems

• Human evaluations commonly measure overall task fulfillment or separately rate desired qualities, with fluency required across NLG tasks and criteria varying by task.Fluency concerns grammar, word choice, and spelling; criteria are not standardized, so the discussion focuses on prominent task-specific criteria. For machine translation, evaluators assess fluency and adequacy, meaning whether the hypothesis represents the reference information.Source: §3.2 · pp. 10–11
• For abstractive summarization, evaluators assess informativeness and coherence, with more elaborate protocols also checking fluency, non-redundancy, referential clarity, focus, and structure.Source: §3.2 · p. 11
• For question answering and question generation, evaluation examines plausibility, correctness, consistency with context, answerability, and relevance to the source material.Source: §3.2 · p. 12
• For data-to-text generation, human judges compare generated text with the data and assess faithfulness to represented facts and informativeness or coverage of important details.Source: §3.2 · p. 12
• Dialogue and image-caption evaluations use broader or task-specific criteria, including engagingness, humanness, persona adherence, emotion consistency, caption relevance, and thoroughness.Source: §3.2 · pp. 12–13
• Human evaluation typically requires multiple skilled evaluators, clear guidelines, and simultaneous assessment of several desired output qualities rather than one score.Source: §3.2 · p. 13

• The survey standardizes discussion with context, reference, and hypothesis notation, while Table 2 lists the corresponding input and output forms for each NLG task.The notation supports multiple references and task-specific contexts such as text, images, tables, and conversation histories.Source: §4 · p. 14, Table 2 · p. 14
• The taxonomy begins with context-free metrics, which compare hypotheses only with references and are task-agnostic, versus context-dependent metrics, which also judge hypotheses against the input context and are typically task-specific.Context-dependent metrics proposed for one task may require substantial adaptation for others with different input types.Source: §4 · p. 14
• Metrics are further classified as trained or untrained: untrained methods use fixed heuristics over words, characters, or embeddings, whereas trained methods learn from human annotations or representations of references, hypotheses, and contexts.Trained metrics may use other metrics as features or learn end-to-end with linear regression, SVMs, or deep neural networks.Source: §4 · p. 14
• Most context-dependent metrics are trained, with only a handful remaining untrained.Source: §4 · p. 14

5 CONTEXT-FREE METRICS

• Context-free metrics compare hypotheses with references without using input context and are categorized as untrained heuristic metrics or trained metrics with learnable components.Untrained metrics are further divided into word-based, character-based, and embedding-based categories.Source: §5 · pp. 16–17, §5.1 · p. 17

5.1.1 Word-based metrics.

• Word-based metrics score lexical or n-gram overlap, or word edits; BLEU, NIST, and METEOR are widely adopted because of their simplicity.These early metrics generally compare hypotheses and references through surface-form matching.Source: §5.1.1 · p. 17
• BLEU computes corpus-level clipped n-gram precision, combines precisions through a geometric mean, and adds a brevity penalty to discourage short hypotheses.Its precision can otherwise be gamed by producing only a few matching words or n-grams.Source: §5.1.1 · p. 17
• NIST3 extends BLEU by weighting rare matched n-grams more heavily, using an arithmetic combination and modifying brevity penalties to reduce length sensitivity.Its information-gain weighting also reduces the chance of gaming the metric with trivial n-grams.Source: §5.1.1 · p. 18
• GTM combines precision and recall in an F-score, emphasizes contiguous matching runs, approximates maximum match size greedily, and showed higher human-judgment correlations than competing metrics.The exact maximum match-size computation is NP-hard for q > 1, motivating its greedy approximation.Source: §5.1.1 · pp. 18–19
• METEOR addresses BLEU’s missing recall and exact-match limitations with F-score evaluation using exact, stemmed, synonym, and paraphrase unigram matches plus fragmentation penalties.METEOR Universal generalizes the approach across languages by automatically building function-word and paraphrase lists from parallel text.Source: §5.1.1 · pp. 19–20
• Word-based metrics include weighted lexical overlap, ROUGE variants, TF-IDF-weighted CIDEr, semantic SPICE [4], SPIDEr [85], and edit-distance families.These metrics respectively emphasize content/function words, recall-oriented overlap, corpus-specific n-gram informativeness, semantic propositions, combined semantic and fluency signals, or edit operations.Source: §5.1.1 · pp. 20–22
• ROUGE measures recall-oriented overlap through n-grams, longest common subsequences, weighted subsequences, or skip-bigrams, but ROUGE-L can equate different semantic implications when matches share the same order and length.ROUGE-W adds a gap penalty, while ROUGE-S uses skip-bigram co-occurrence and an F-score.Source: §5.1.1 · pp. 20–21
• SPICE [4] compares scene-graph tuples for semantic faithfulness, whereas CIDEr [162] weights reference-supported n-grams with TF-IDF and SPIDEr [85] linearly combines SPICE and CIDEr.SPICE uses object, relation, and attribute tuples with WordNet synonym matching; its reliance on parsing and assumption of fluency can reward grammatically incorrect captions.Source: §5.1.1 · p. 22
• WER-based metrics count insertions, deletions, substitutions, and possible transpositions, while TER, ITER, PER, and CDER address reordering or multiple-reference limitations.WER heavily penalizes different word orders, motivating shift, stem-matching, position-independent, or block-reordering operations.Source: §5.1.1 · pp. 22–23

5.1.2 Character-based metrics.

• Character-based metrics usually do not require tokenization and improve evaluation of morphologically rich languages; characTER applies character-level Levenshtein distance with hypothesis-length normalization.This normalization is empirically shown to correlate better with human judgements.Source: §5.1.2 · p. 23
• chrF compares character n-grams up to order 6, averaging precision and recall; chrF+ adds word unigrams, while chrF++ adds word unigrams and bigrams.The recall weight is controlled by β, and chrF has been adopted for machine translation, image captioning, and summarization.Source: §5.1.2 · pp. 23–24

5.1.3 Embedding based metrics.

• Character- and word-based metrics rely largely on surface matches, so they often miss semantic similarities such as synonymy or relatedness.Embedding-based metrics instead compare similarities between hypothesis and reference word embeddings trained on large corpora, including Word2Vec [108] and GloVe.Source: §5.1.3 · p. 24
• Embedding metrics include greedy token matching, sentence-vector cosine similarity, dimension-wise extreme vectors, and WMD’s optimal transport between word embeddings.Greedy Matching averages bidirectional matches to ensure symmetry, while WMD uses Euclidean-distance-based optimal matching and can partially map words across references.Source: §5.1.3 · pp. 24–25
• Embedding-based metrics address ordering and semantic structure through WMD_O, WE_WPI, MEANT, and MEANT2.0, which incorporate position, semantic roles, lexical similarity, IDF, and n-grams.WMD_O penalizes different word ordering; WE_WPI uses word-position information; MEANT scores semantic frames and role fillers, while MEANT2.0 emphasizes content words and word order.Source: §5.1.3 · pp. 25–26
• YiSi-1 is an F-score that computes n-gram similarity as an aggregate of weighted word-embedding cosine similarity, optionally accounting for shallow semantic structure; YiSi-0 is resource-free, using longest common character-substring accuracy.Source: §5.1.3 · pp. 26–27
• BERTScore correlates better with human judgments for image captioning, while MoverScore shows competitive correlations across four NLG tasks.BERTScore uses greedy one-to-one token matching with contextualized embeddings; MoverScore uses soft or partial many-to-one alignments over words or n-grams.Source: §5.1.3 · p. 27

5.2 Trained metrics

• Trained metrics contain learnable components and divide into feature-based metrics using heuristic inputs and end-to-end metrics trained directly on hypothesis-reference sentences.Feature-based inputs include n-gram precision, recall, BLEU, or METEOR, while learnable models range from linear regressors to deep neural networks.Source: §5.2 · p. 27, §5.2.1 · pp. 27–28

Linear Regression

• BEER uses precision, recall, and F1-score over character n-grams and word-level unigrams, with unigram statistics separated by function and content words.The unigram statistics are also computed over all words collectively.Source: Linear Regression · p. 28
• Permutation-tree features evaluate word order or fluency alongside the lexical statistics.Source: Linear Regression · p. 28
• The BEER model is a simple linear function whose feature values are computed from a hypothesis and reference, with weights learned by linear regression from WMT13 human judgments.Source: Linear Regression · p. 28

SVM Regression

• BLEND combines existing untrained metrics with an SVM regressor to improve correlation with human judgments.It uses 57 metric scores as features and human direct-assessment scores from WMT15[149] and WMT16[14] as the gold-standard target.Source: SVM Regression · p. 28
• The 57 features are organized into lexical, syntactic, and semantic metric categories.They include 25 lexical metrics, 17 syntactic metrics from the Asiya toolkit, and 13 semantic metrics covering named entities, semantic roles, and discourse representation.Source: SVM Regression · p. 28
• The ablation study found that combining all three metric categories provides the best results.The analysis tested the contribution of each category to the combined metric.Source: SVM Regression · p. 28

Grid search with bagging

• Q-Metrics [111] targets better correlation with human judgements on question-generation answerability than existing n-gram metrics.Source: Grid search with bagging · p. 28
• The method assigns unequal weights to function words, Wh-words, named entities, and content words instead of treating all n-grams equally.Source: Grid search with bagging · p. 28
• It computes category-specific n-gram precision and recall, then combines them through weighted averages P_avg and R_avg.Source: Grid search with bagging · p. 28
• Grid search and bagging tune the weights and δ for BLEU, NIST, METEOR, and ROUGE to maximize correlation with human scores.Source: Grid search with bagging · p. 29

Neural networks/ Deep Learning

• Sharif et al. train a feedforward network on METEOR, CIDEr, WMD, and SPICE features to classify captions as machine- or human-generated, while NNEval adds BLEU(1-4).Training uses Flicker30k references as positive samples and captions generated by Show and Tell, Show, Attend and Tell, and Adaptive Attention [97] as negatives.Source: Neural networks/ Deep Learning · p. 29

5.2.2 End-to-end Trained metrics.

• End-to-end trained metrics use neural networks to learn directly from hypothesis–reference sentences, including feed-forward, recurrent, and recently pretrained-transformer models.These models use static or contextualized word embeddings, with transformer models appearing in newer metrics.Source: §5.2.2 · p. 29
• SIMILE trains a sentence encoder on ParaNMT paraphrase pairs, scores cosine similarity, and adds a length penalty to discourage repetitive long generations.The length-penalty influence is tuned over 0.25 and 0.5.Source: §5.2.2 · p. 29
• ESIM, adopted for translation evaluation by Mathur et al., computes sentence representations with a trained BiLSTM and compares them using cross-sentence attention.The metric is trained on Direct Assessment human-evaluation data collected for WMT 2016.Source: §5.2.2 · pp. 29–30

Transformer-based metrics

• Transformer architectures use feed-forward networks with attention, and pretrained BERT, RoBERTa, and XLNet models have entered trained NLG evaluation metrics.These models replace the established reliance on recurrent neural networks for NLP tasks.Source: Transformer based trained metrics · p. 30

BERT for MTE

• BERT for MTE jointly fine-tunes BERT on concatenated reference–hypothesis pairs, using an MLP over the [CLS] representation to produce scores and improving correlations over RUSE.Unlike RUSE, it pair-encodes the candidate and reference sentences together and jointly fine-tunes the pretrained encoder for evaluation.Source: Transformer based trained metrics · p. 30

BLEURT

• BLEURT [138] pretrains BERT on synthetically perturbed sentence pairs using metric, back-translation, entailment, and perturbation-origin signals, then achieves state-of-the-art performance on WMT and WebNLG.The signals are aggregated with a weighted sum, and the model is fine-tuned on task-specific supervised data with regression loss.Source: Transformer based trained metrics · pp. 30–31

NUBIA

• NUBIA [68] uses a 3-stage architecture combining transformer-based sentence-similarity and logical-inference features, GPT-2 perplexity for grammatical correctness, and an aggregator that is linear regression or feed-forward neural.Source: Transformer based trained metrics · p. 31

6 CONTEXT-DEPENDENT METRICS

• Context-dependent metrics incorporate input context, with untrained and trained categories, but their task-specific design limits reuse across different NLG tasks.This contrasts with context-free metrics such as BLEU and ROUGE, which are used across a wide range of tasks.Source: §6 · p. 31

6.1 Untrained metrics

• Untrained context-dependent metrics use either word-based or embedding-based features from the hypothesis and its input context.Word-based metrics include ROUGE-C and PARENT, while YiSi-2 is an embedding-based metric discussed in this section.Source: §6.1 · p. 32, §6.1.1 · p. 32, §6.1.2 · p. 33

6.1.1 Word based metrics.

• ROUGE-C compares a candidate summary with the document rather than a reference summary, making it especially beneficial when a reference summary is unavailable, including query-focused summarization.Source: §6.1.1 · p. 32
• PARENT evaluates data-to-text outputs by matching hypothesis n-grams to both references and table records through entailment probabilities.Its entailed precision and recall are aggregated across n-grams, and the final instance score is the F-score of their combined values.Source: §6.1.1 · pp. 32–33

6.1.2 Embedding based metrics.

• YiSi-2 is a bilingual, reference-less MT quality-estimation metric that uses multilingual BERT cross-lingual embeddings to compare source input with MT output.It can optionally incorporate a shallow semantic parsing module.Source: §6.1.2 · p. 33

6.2 Trained metrics

• Trained context-dependent metrics use neural architectures ranging from MLPs to Transformers, including dialogue metrics such as ADEM and RUBER and the image-captioning metric LEIC.LEIC addresses the limitations of overlap-based caption metrics by using the image to distinguish human from machine-generated captions; RUBER avoids requiring large-scale human scores by combining referenced and unreferenced evaluation.Source: §6.2 · p. 33, §6.2.1 · p. 33

6.2.1 End-to-end Trained metrics.

• SSREM [6] combines reference-based and unreferenced scores, using GloVe embeddings for context-response relevance and sentence mover’s similarity with ELMo embeddings [123] for response-reference similarity.The two scores are combined into the final metric formulation.Source: §6.2.1 · p. 35
• SSREM [6] uses speaker-sensitive samples rather than one random negative, drawing hard negatives from four speaker and conversation relationships.The cases range from the same speaker in the same conversation to a random other speaker, ordered by hypothesized similarity to the reference.Source: §6.2.1 · p. 35
• These speaker-sensitive cases are empirically more challenging to distinguish than randomly sampled negatives, so SSREM trains the reference above all four hard-negative responses.Training uses the Twitter corpus because it contains multiple conversations per speaker, enabling speaker-sensitive sampling.Source: §6.2.1 · p. 35

RUBER with BERT embeddings

• Ghazarian et al. [51] enhance RUBER by replacing word2vec embeddings with BERT embeddings, while training only the MLP classifier on DailyDialog.The unreferenced model pools BERT embeddings, uses a binary MLP classifier with cross-entropy loss, and does not fine-tune BERT.Source: Transformer based metrics · p. 36

MaUde

• MaUde [146] scores dialogue responses without references by combining BiLSTM-based context and response representations in a sigmoid classifier.It is trained on PersonaChat using syntactic and semantic negative responses, including shuffled, corrupted, randomly sampled, generated, and back-translated responses; a back-translated reference is also positive.Source: Transformer based metrics · p. 36

RoBERTa-eval

• RoBERTa-eval encodes the dialogue context and proposed response into a single vector using RoBERTa, then applies an MLP classifier to produce a score from 1 to 5.Its RoBERTa parameters can be finetuned and its MLP parameters are learned during training; scaling keeps the score in the range of 1 to 5. It is trained on DailyDialog using human response-quality annotations and random negative responses to obtain varied quality.Source: Transformer based metrics · pp. 36–37

Poor correlations

• Automatic metrics often correlate poorly with human judgments, with reliability generally higher at the system level than at the sentence level.BLEU, NIST, SSA, GTM, and LSA correlated negatively with human fluency judgments and showed moderate to low correlation with adequacy; correlations are unreliable when ranking similarly capable systems.Source: §7 · pp. 37–38

Uninterpretability of scores

• Automatic metrics provide a single score that does not reveal which human evaluation criterion—such as fluency, informativeness, or coherence—it represents.BLEU can assign identical scores to incorrect n-gram permutations, penalize valid translations that differ from references, and fail to indicate progress reliably.Source: §7 · p. 38

Bias in the metrics

• Reported biases include BLEU favoring n-gram-based over rule-based systems, GAN evaluators generalizing poorly beyond their training systems, and ADEM collapsing dialogue scores into a narrow median-centered range [135].ADEM’s response encodings also show high conicity, with little separation between relevant and irrelevant responses.Source: §7 · p. 38

Poor adaptability across Tasks

• Metrics developed for one NLG task may not transfer appropriately to another because evaluation criteria differ, exemplified by BLEU and METEOR’s unsuitable n-gram overlap for dialogue [70] [84] [111].A valid dialogue response may share no n-grams with the reference while remaining correct for the context.Source: §7 · p. 39

Inability to capture all nuances in a task

• Even task-specific metrics fail to capture key nuances, including factual consistency in abstractive summaries [72], faithfulness in data-to-text generation, dialogue-response diversity [84], and linguistic divergence in Hindi-English translation.The translation divergences include lexical, categorical, pleonastic, and stylistic differences.Source: §7 · p. 39

8 EVALUATING EVALUATION METRICS

• Automatic metrics are commonly assessed by correlating their scores on generated or synthetic outputs with human judgments under task-specific criteria.This methodology aims to determine how well an automatic metric serves as a proxy for human evaluation.Source: §8 · p. 39

8.1 Correlation Measures

• Pearson measures linear dependence, whereas Spearman measures monotonic dependence through ranks and Kendall’s τ uses concordant and discordant observation pairs.Pearson assumes linearity and bivariate normality; all three coefficients range from -1 to +1.Source: §8.1 · pp. 39–40

8.2 Significance Tests:

• A Pearson correlation of 0.7 from 5 samples may arise by chance, so coefficients do not establish significance; p-values accompany tests, but separate p-values cannot determine whether two correlations differ significantly.Source: §8.2 · pp. 41–42

8.3 Issues and Influence of the Data and Human-evaluation setup

• Correlation scores depend on references, human judges, corpus quality, dataset characteristics, and noise, so removing outliers and other biases can improve robustness.Large numbers of good-quality references yield better correlations, whereas automatically or heuristically collected references may not be reliable gold standards.Source: §8.3 · p. 42
• Automatic metrics reliably assess fluency, but they disagree with human evaluations on adequacy and correctness, allowing seq2seq models to score highly despite underperforming rule-based models.The section also notes varied rankings between automatic and human evaluations and the effects of high-quality, low-variance human judgments on metric correlations [19].Source: §8.3 · pp. 42–43

8.4 Methods to Evaluate Robustness of Metrics

• Robustness evaluations use automated text manipulations and adversarial examples to expose metric failures in corner cases.Examples include BLEU permutation attacks that preserve matching n-gram counts and therefore scores despite producing potentially nonsensical reordered sentences.Source: §8.4 · p. 43
• Image-caption robustness tests replace people or scenes, switch roles, or use related captions, then check whether metrics detect the changed content [70].These tests create distracted captions and evaluate whether metrics correctly identify altered scenes or people.Source: §8.4 · p. 43
• Other robustness methods test trained evaluators on word-order reversal, punctuation or stopword removal, part-of-speech combinations, and simplified scenarios, including Evaluator Reliability Error (ERE) [76].Source: §8.4 · p. 43

Developing a common code base for evaluation metrics

• A common code base would broaden metric use, improve cross-paper comparability, ease baseline comparisons, and enable code-based scrutiny including adversarial testing.The survey notes that many studies report only easily available metrics and that shared implementations could support white-box attacks and carefully crafted adversarial examples.Source: §9 · p. 43

Building datasets containing human judgements

• Automatic evaluation metrics rely on datasets containing tuples of context, reference response, proposed response, and human-assigned scores, where proposed responses may be human-, synthetically-, or NLG-generated.These datasets provide data for training and evaluating automatic metrics, with proposed responses evaluated by humans and assigned scores.Source: §9 · p. 44

Developing task-specific context-dependent metrics

• Task-specific context-dependent metrics are needed because reference overlap alone cannot adequately assess coherence, factual consistency, or relevance in dialogue, summarization, question answering, and data2text generation.Most existing metrics are context-free, while context-dependent metrics have focused largely on dialogue; many valid responses need not share substantial word or semantic overlap with references.Source: §9 · p. 44

Developing more interpretable metrics

• More interpretable metrics should report separate criterion scores rather than one opaque overall score, making system weaknesses actionable and comparisons more informative.Separate fluency and adequacy scores could guide developers toward language-model or translation-model improvements and help users or reviewers weigh trade-offs between systems.Source: §9 · p. 44

Creating robust benchmarks for evaluating evaluation metrics

• Robust metric benchmarks should use adversarial responses to test relevance, factuality, robustness, and biases, while also scrutinizing whether correlation measures are reliable.The survey highlights possible training-system and pretraining-domain biases, including GAN evaluators favoring seen systems and social-media-pretrained metrics disadvantaging formal language; prior work is cited for QA and dialogue.Source: §9 · pp. 44–45

• Reliable automated evaluation remains difficult because NLG systems involve diverse tasks and criteria for judging generated hypotheses.The field needs robust metrics that accurately quantify progress despite its rapid evolution.Source: §10 · p. 45
• The survey classifies metrics as context-free or context-dependent, with trained and untrained variants using word-, character-, or embedding-based information.This taxonomy organizes task-agnostic and task-specific metrics reviewed in the survey.Source: §10 · p. 45
• Most existing metrics are context-free, revealing a major need for task-specific context-dependent metrics because context is important in many NLG tasks.Ignoring task context is therefore a substantial gap in current evaluation methods.Source: §10 · p. 45
• Evaluation requires appropriate correlation measures and significance tests because commonly used measures can be unreliable in some situations.For example, Pearson correlation is mostly appropriate when a linear relationship between variables is expected.Source: §10 · p. 45
• Existing metrics have been criticized for weak human correlation, uninterpretability, inherent biases, poor cross-task adaptability, and failure to capture task-specific nuances.These shortcomings motivate continued development and critical examination of evaluation methods.Source: §10 · p. 45
• The survey recommends common code bases, fine-grained task-specific human-judgement datasets, context-dependent metrics, interpretable scores, and robust benchmarks for exposing shortcomings and biases.These directions aim to improve reproducibility, metric adoption, actionable feedback, and critical validation.Source: §10 · p. 45