Evaluation of Text Generation: A Survey

1. Introduction

• NLG systems transform structured, linguistic, or visual inputs into coherent human-understandable text across tasks including dialogue, translation, storytelling, and summarization.Because many tasks permit multiple valid outputs, human evaluation remains the standard, although its expense drives reliance on automatic metrics for progress measurement and system optimization.Source: §1 · p. 1

1.3 Outline of The Survey

• The survey organizes evaluation into human-centric, untrained automatic, and machine-learned metrics, then examines shared tasks, summarization, long-text generation, and future research directions.Human methods use judges or Turing tests; untrained metrics compare outputs with references using overlap, distance, or diversity; machine-learned metrics act as digital judges.Source: §1 · p. 2, §1.3 · pp. 4–5

1.1 Evolution of Natural Language Generation

• NLG evolved from template-based and rule-driven pipelines toward data-driven representation learning and neural architectures, including sequence-to-sequence models, attention, pointers, and Transformers.Large datasets, treebanks, corpora, and shared tasks supported this progression, while modern neural systems address tasks such as summarization, translation, dialogue, paraphrasing, and long-text generation.Source: §1.1 · pp. 2–3

1.2 Why a Survey on Evaluation of Natural Language Generation

• The survey addresses a gap in prior task- and model-specific reviews by providing an extensive, up-to-date analysis of evaluation metrics for neural text generation systems.Earlier surveys covered NLG broadly or reviewed particular tasks, models, or human-centric evaluation, while this work is dedicated to NLG evaluation.Source: §1.2 · p. 4

• Human evaluations are considered the most important NLG evaluation method and the gold standard for developing automatic metrics because they assess value to people.Automatic metrics still fall short of replicating human decisions, and 20 of 26 ACL2018 generation papers reported human evaluation results.Source: §2 · p. 5
• Human evaluations are expensive and time-consuming, particularly for tasks requiring domain expertise, while crowdsourcing introduces quality-control problems.Platforms such as Amazon Mechanical Turk enable larger-scale, lower-cost experiments but create their own challenges.Source: §2 · p. 5
• Some generated-text qualities, including diversity, are poorly suited to human evaluation even with many annotators.Source: §2 · p. 5
• Inconsistent human-evaluation procedures and incomplete reporting hinder reproducibility and comparison across systems; only 57% of sampled ACL and INLG papers reported participant numbers.Source: §2 · p. 5
• Human-judgment approaches are organized into intrinsic and extrinsic evaluations, while using human judgments to train models is addressed separately.Source: §2 · p. 5

Human evaluation dimensions

• Human evaluators score generated texts for overall quality or dimensions such as fluency, coherence, correctness, adequacy, and factuality.Intrinsic evaluation commonly presents multiple model samples for individual quality judgments.Source: §2.1 · pp. 6–7

Comparative evaluation

• Comparing outputs through preference or ranking can improve inter-annotator agreement, but it measures relative rather than absolute quality.RankME (Novikova et al., 2018) adds magnitude estimation to indicate how much better one chosen text is than alternatives.Source: §2.1 · p. 6
• Best-worst scaling simplifies costly or complex comparisons among many candidates while retaining reliable comparative results.It asks participants to select the best and worst elements rather than fully rank a set, and has been used in NLG tasks (Koncel-Kedziorski et al., 2019).Source: §2.1 · p. 7

Human evaluation dimensions

• Over 70% of generated single-sentence summaries contained hallucinations across several modeling approaches, underscoring factuality as a key evaluation dimension.Other content-oriented dimensions include commonsense, logic, coherence, and consistency with the provided context.Source: §2.1 · p. 7

Standardization and limitations

• ˜25% of generation papers failed to identify evaluation dimensions, while fewer than half defined them, motivating standardized platforms such as GENIE and GEM (Gehrmann et al., 2021).Standardization can improve comparability but may introduce rigidity and encourage overfitting to inappropriate objectives.Source: §2.1 · p. 8

Definition and motivation

• Extrinsic evaluation measures how successfully an NLG system performs in a downstream task, making it meaningful but expensive and difficult to conduct.Intrinsic evaluations are therefore more common, partly because recent NLG research has shifted toward subtasks rather than full systems.Source: §2.2 · p. 8

User success

• User-success evaluations measure what people can take away from generated text, such as improved decision making, comprehension, or fewer mistakes following instructions.Young (1999) evaluated automatically generated instructions by counting subjects’ mistakes.Source: §2.2 · p. 8

System success

• System-success evaluations measure whether an NLG system fulfills its designed purpose, including recipients quitting smoking after personalized letters or the amount of post-editing required.Reiter et al. (2003) used smoking cessation as the downstream outcome, while post-editing measures how many changes people make to machine-generated text.Source: §2.2 · p. 9

Dialogue-system evaluation

• Dialogue-system evaluations commonly collect human feedback through conversation length, user ratings, or extended interaction with operational systems, but crowdsourcing remains most common.The Alexa Prize gathers feedback from real users over several months, while AMT studies can produce quality metrics comparable to evaluations involving direct system interaction.Source: §2.2 · p. 9

• Most fluency-focused evaluations require only proficiency in the generated text’s language, and broad-audience systems benefit from recruiting evaluators as widely as possible.Source: §2.3 · p. 9
• In-person evaluations facilitate participant training, detailed feedback, and control over targeted recruitment, but are costly, time-consuming, smaller, less representative, and vulnerable to response bias.Source: §2.3 · p. 9
• Online crowdsourcing enables large-scale, time-efficient, inexpensive evaluations with broader geographic reach, but quality control, demographic skew, and mismatch with end users remain concerns.Researchers may recruit paid workers through Amazon Mechanical Turk or incentivized participants through LabintheWild.Source: §2.3 · pp. 9–10
• Specialized evaluators are necessary when tasks target particular users, require domain expertise, or involve long or multiple documents.For citation generation, Luu et al. (2020) used NLP researchers because rating citations requires understanding two scientific documents and academic writing conventions.Source: §2.3 · p. 10

2.4 Inter-Evaluator Agreement

• Agreement analysis appears in only 18% of 135 generation papers, and reported agreement is usually below commonly acceptable levels despite high agreement not always being the right goal.High agreement can indicate a well-defined task and consistently noticeable differences, whereas low agreement may indicate poorly defined tasks or unreliable differences; however, natural-language richness makes maximal agreement undesirable in some cases.Source: §2.4 · pp. 10–11

2.4.1 Percent agreement

• Percent agreement reports the proportion of generated texts receiving identical evaluator scores, but it ignores agreement that could occur by chance.Its value ranges from 0 when evaluators never agree to 1 when they agree on every text.Source: §2.4.1 · p. 11

2.4.2 Cohen’s κ

• Cohen’s κ extends percent agreement by estimating and removing the probability that two evaluators agree by chance.The chance probability is estimated from each evaluator’s frequency of assigning the possible scores.Source: §2.4.2 · p. 11

2.4.3 Fleiss’ κ

• Fleiss’ κ generalizes chance-corrected agreement to multiple evaluators by calculating agreement across all evaluator pairs.It estimates chance agreement from score frequencies across all annotators before applying the κ calculation.Source: §2.4.3 · p. 12

2.4.4 Krippendorff’s α

• Krippendorff’s α measures weighted evaluator disagreement, allowing some score disagreements to be penalized more heavily than others.It assigns weights to score pairs, with identical scores receiving zero disagreement weight, and is the most common weighted measure in surveyed NLG papers.Source: §2.4.4 · p. 12

Motivation

• Automatic metrics are used because human evaluation is costly, time-consuming, and not always repeatable, despite humans often evaluating generated text with little effort.They support both new-model development and comparison with state-of-the-art approaches.Source: §3 · p. 13

Overview

• Untrained automatic metrics provide fast, efficient scores for comparing generated text with human-written reference text during NLG model development.They are used across tasks including machine translation, image captioning, and question generation.Source: §3 · p. 13

Metric categories

• The metrics are grouped into n-gram overlap, distance-based, diversity, content overlap, and grammatical feature-based categories.Grammatical feature-based evaluation generally relies on syntactic parsing rather than a single defined metric.Source: §3 · p. 13

Limitations

• Limited human studies report that most untrained automatic metrics correlate more weakly with human judgments, with correlation strength depending on the evaluator.The survey therefore discusses the advantages and drawbacks of commonly used metrics and clusters some by efficiency criteria in Tables 1 and 2.Source: §3 · pp. 13–14

3.1 n-gram Overlap Metrics for Content Selection

• n-gram overlap metrics compare machine-generated text with human-authored references using features such as precision, recall, F-scores, and weighted n-gram matching.The f1-score balances precision and recall harmonically, while bleu uses a weighted geometric mean of n-gram precision scores and was originally proposed for machine translation.Source: §3.1 · p. 14
• bleu can support machine-translation benchmarking but may correlate poorly with human judgments in other generation tasks and overlook semantic or global coherence.Perfect bleu scores have been associated with grammatically correct text lacking semantic or global coherence, and outliers can create misleadingly high correlations that disappear when removed.Source: §3.1 · pp. 14–15
• rouge emphasizes recall and offers interpretable variants for unigram, bigram, longest-subsequence, and skip-bigram overlap, while enabling significance testing of system-level differences.Its n-gram reliance still leaves narrative flow, grammar, topical flow, and factual correctness unmeasured, especially in long-text generation.Source: §3.1 · p. 16
• meteor extends overlap matching with recall-weighted unigram precision, stemming, and synonym matching, producing good sentence- or segment-level correlation with human judgments.Unlike bleu, meteor is designed for sentence-level rather than corpus-level comparison and weights recall more heavily than precision.Source: §3.1 · p. 16
• Other untrained metrics target specialized needs, including cider’s consensus-based image-caption evaluation, nist’s weighting of rare n-grams, hlepor’s morphological and syntactic features, and ribes’s word-order sensitivity.Cider can correlate well with human judgments on some datasets, although semantic-content metrics such as spice have sometimes correlated better (Anderson et al., 2016; Liu et al., 2017).Source: §3.1 · pp. 16–17

3.2.1 Edit Distance-Based Metrics

• Edit-distance metrics quantify the minimum operations needed to transform one text into another, but WER is unbounded and weakly correlated with human or retrieval-system performance, while TER relies on exact matches to a single reference.TER normalizes edits by average reference length and can partly address its narrow coverage through lattices of reference translations.Source: §3.2.1 · p. 18

3.2.2 Vector Similarity-Based Evaluation Metrics

• Embedding-based metrics capture semantic similarity beyond exact overlap: WMD is interpretable and hyperparameter-free but becomes costly and loses sentence-level relations on long documents.MEANT and YiSi incorporate semantic frames, role fillers, distributional lexical similarity, or shallow semantic structures, while WMD combines bag-of-words frequencies with word-embedding similarity and has supported sentence-level NLG tasks.Source: §3.2 · p. 18, §3.2.2 · p. 19
• SMD correlated better with human judgments than ROUGE on summarization, extending WMD-style distances from words to length-weighted sentence embeddings.SMD measures the cumulative distance needed to move sentence embeddings from one document to match another; a newer version achieved higher human-judgment correlation using word, sentence, ELMO, and BERT embeddings.Source: §3.2.2 · p. 20

• Lexical diversity is valuable for several NLG tasks, but diversity metrics are rarely sufficient alone because diverse text may reduce quality or conflict with task requirements.Weather forecasting illustrates that consistent expression can be preferred over greater lexical variety.Source: §3.3 · p. 20
• Type-token ratio divides unique words by total words, providing an intuitive measure of vocabulary richness.Source: §3.3 · p. 20
• TTR decreases as texts lengthen because additional tokens are less likely to introduce new types; HD-D was proposed to compare texts of different lengths.Source: §3.3 · pp. 20–21
• N-gram repetition metrics generalize TTR, and Li et al. (2016) used BLEU plus distinct unigram and bigram counts to evaluate diversity-promoting dialogue objectives.Modeling mutual information between sources and targets reduced bland responses and improved response diversity.Source: §3.3 · p. 21
• Self-BLEU averages each generated sentence’s BLEU score against the others as references, with lower scores indicating higher diversity, but its effectiveness is disputed.Reports find good diversity evaluation, alongside weaknesses in generating diverse output and detecting GAN mode collapse (Zhu et al., 2018; Chen et al., 2018a; Lu et al., 2018; Caccia et al., 2018; Goodfellow et al., 2014).Source: §3.3 · p. 21

3.4 Explicit Semantic Content Match Metrics

• Explicit semantic content metrics compare human and generated texts using semantic or conceptual information units beyond n-grams, with generally strong correlation to human judgments.These metrics define similarity at semantic and conceptual levels rather than relying only on surface overlap.Source: §3.4 · p. 21
• Pyramid identifies weighted summarization content units from annotated equivalent spans, while PEAK automates weight assignment and correlates well with human judgments.The original pyramid method remains difficult to automate because it requires manual human labeling.Source: §3.4 · p. 21
• SPICE parses captions into scene graphs of objects, attributes, and relations, computes proposition-level F-scores, and captures broader semantic similarity than n-gram matching.SPICE correlates strongly with human ratings but ignores fluency; SPIDEr combines SPICE with CIDEr to address syntactic shortcomings, although its human correlation is unreported.Source: §3.4 · pp. 21–22, Figure 3 · p. 22

3.4.1 Semantic Similarity Models used as Evaluation Metrics

• Semantic similarity models evaluate task-level meaning through sentence-level methods including STS and paraphrase identification, with applications across translation, summarization, dialogue, and captioning.STS uses weighted Pearson correlation against human similarity judgments, while paraphrase identification seeks a reference paraphrase closer in wording to the hypothesis; tiger applies this idea to image captions.Source: §3.4.1 · pp. 22–23
• Textual entailment tests whether a hypothesis follows from a premise, while machine comprehension matches passages to questions and identifies answer regions for generation-task evaluation.These methods have been applied to machine translation, summarization, language modeling, video captioning, question generation, and document summarization.Source: §3.4.1 · p. 23

3.5 Syntactic Similarity-Based Metrics

• Syntactic similarity metrics compare reference and hypothesis texts through their overall grammatical or sentence structure.They capture similarity at a structural level rather than only through surface word overlap.Source: §3.5 · p. 23
• POS-tag-based metrics have been applied to machine translation, story generation, summarization, and question generation.Examples include tesla, which combines bilingual phrase-table synonyms with POS tags, and approaches using POS n-grams with morphemes and lexicon probabilities.Source: §3.5 · pp. 23–24
• Syntactic analyses enrich evaluation by representing sentence structure with constituent labels, head-modifier dependencies, shallow parsers, or dependency trees.These representations have been used in machine translation evaluation and other text-generation settings.Source: §3.5 · p. 24

4. Machine-Learned Evaluation Metrics

• Machine-learned evaluators address diverse NLG outputs where word-overlap metrics fail by modeling qualities such as factual correctness, naturalness, fluency, and coherence.Such models are especially useful when plausible outputs share no words with references and embedding similarity is insufficient.Source: §4 · p. 24

4.1 Sentence Semantic Similarity Based Evaluation

• Sentence-semantic metrics encode generated and reference sentences into embeddings, progressing from unsupervised and weighted word-vector models to contextualized BERT and RoBERTa encoders.Earlier models trade contextual richness, word order, training speed, and computational cost, while Sentence-BERT and ESIM adapt representations for broader evaluation use.Source: §4.1 · pp. 24–26

4.2 Regression-Based Evaluation

• RUSE treats machine-translation evaluation as regression over hypothesis-reference representations and correlates with human judgments at both segment and system levels.It extracts relations through concatenation, element-wise products, and absolute element-wise distance between GRU-based representations.Source: §4.2 · p. 26

4.3 Evaluation Models with Human Judgments

• Human-judgment models target qualities missed by overlap metrics, but ADEM correlates with human scores while conservatively underrating plausible responses, whereas HUSE detects low-diversity generations.HUSE compares model and human generation distributions using statistical and human-evaluation signals, addressing diversity in dialogue and story generation.Source: §4.3 · pp. 27–28

4.4 BERT-Based Evaluation

• BERT-based evaluators improve semantic and logical assessment: BERTScore correlates with human judgments, RoBERTa-based models outperform BLEU and ROUGE in robustness, and comparator evaluation correlates highly with humans.BLEURT uses synthetic pre-training to improve robustness to quality drift, while the comparator learns pairwise win/loss relations and latent skill ratings.Source: §4.4 · pp. 29–30

4.5 Evaluating Factual Correctness

• Factuality metrics target generation errors, including up to 30% factual inconsistencies in abstractive summaries, using entailment, alignment, reinforcement learning, and question answering.Localized dependency-arc entailment improves factual consistency; PARENT correlates better with human judgments than several baselines; APES measures answer retrieval for named-entity questions.Source: §4.5 · pp. 31–33

4.6 Composite Metric Scores

• Composite metrics combine multiple signals to capture adequacy, fluency, and diversity, with image-captioning approaches reported to correlate well with human judgments and achieve state-of-the-art performance.These approaches include combinations of METEOR, WMD, and SPICE and an rAIRL-based reward that decouples word-level rewards.Source: §4.6 · p. 33

5. Shared Tasks for NLG Evaluation

• Shared tasks efficiently benchmark NLG systems with common data and metrics, while supporting new tasks amid limited evaluation pipelines and raising organizational and ethical concerns.They encourage progress toward the state of the art, but variability and limited standardization can create problems (Parra Escartín et al., 2017).Source: §5 · p. 34

5.1 Generating Referring Expressions; 5.4 Regular Expression Generation from Attribute Sets

• The GRE Challenge evaluated content determination through minimal, unique, and humanlike referring expressions, while the tuna Challenge covered attribute selection, realization, and end-to-end generation (Gatt & Belz, 2008).The tuna task selects attributes that uniquely distinguish an intended referent from distractors.Source: §5.1 · p. 35, §5.4 · p. 35

5.2 Embedded Text Generation

• GIVE evaluated situated instruction generation across referring expressions, aggregation, grounding, realization, and user modeling by guiding human players through simulated 3D environments.The challenge was organized across four consecutive years.Source: §5.2 · p. 35

5.3 Regular Expression Generation (REG) in Context

• GREC challenges mapped intended referents to clear, fluent, coherent text and evaluated coreference-chain selection for people in encyclopedic texts.GREC-Full focused on replacing referring expressions in context, whereas GREC-NEG selected correct coreference chains.Source: §5.3 · p. 35

5.5 Deep Meaning Representation to Text (SemEval)

• SemEval Task 9 evaluated generation of valid English sentences from Abstract Meaning Representation structures.SemEval workshops introduce annual shared tasks to advance semantic analysis and create annotated datasets.Source: §5.5 · p. 36

5.6 WebNLG

• WebNLG benchmarked RDF-triple-to-text systems on microplanning capabilities including referring expressions, aggregation, lexicalization, realization, and segmentation.Its 2020 challenge expanded dataset size, verbaliser coverage, categories, and language support.Source: §5.6 · p. 36

5.7 E2E NLG Challenge

• The E2E NLG Challenge provided a large, high-quality dataset for spoken-dialogue response generation, requiring joint sentence planning and surface realization without costly alignment (Dusek et al., 2018).The challenge targeted response generation models in spoken dialogue systems.Source: §5.7 · p. 36

5.8 Data-to-Text Generation Challenge; 5.9 GEM Benchmark

• Data-to-text challenges benchmarked content realization from coherent relation sets, while GEM continuously adds datasets, metrics, and human-evaluation standards for testing diverse NLG tasks and strategies (Banik et al.; Gehrmann et al., 2021).Earlier data-to-text work often introduced datasets and benchmarks without organizing challenges.Source: §5.8 · p. 36, §5.9 · pp. 36–37

6.1 Automatic Document Summarization Evaluation

• Automatic summarization evaluation combines reference-independent, document-closeness, and reference-summary-closeness criteria within intrinsic and extrinsic methods.Intrinsic evaluation considers content, quality, and factual consistency, whereas extrinsic evaluation measures effects on downstream tasks such as relevance assessment, reading comprehension, and question answering.Source: §6.1 · p. 37, §6.1.2 · p. 40

6.1.1 Intrinsic Methods

• Intrinsic summarization evaluation covers content overlap, coherence, readability, fluency, focus, informativeness, and factual consistency, but quality measurement remains an open challenge.ROUGE correlates less with human judgments for topic-focused meeting summarization, while factual faithfulness is especially important because generated summaries frequently contain unsupported claims.Source: §6.1.1 · pp. 38–39

6.1.2 Extrinsic Summarization Evaluation Methods

• Extrinsic summarization metrics evaluate summaries through downstream relevance, reading-comprehension, and question-answering performance, with SERA correlating better than ROUGE for scientific article summarization.SERA measures relevance between generated and human-written summaries, while other methods use questions about salient source entities.Source: §6.1.2 · p. 40

6.2 Long Text Generation Evaluation

• Long-text generation is difficult because models may produce locally fluent text that becomes incoherent, changes topics, or violates narrative structure as generation length increases.Grover may restart a story mid-document, whereas PlotMachines more closely follows a beginning-middle-ending structure; GPT-2 can similarly wander as text grows longer.Source: §6.2 · p. 41

6.2.1 Evaluation via Discourse Structure

• Discourse-based evaluation measures topical flow, entity and event structure, and narrative relations, using connective-translation accuracy or similarity between source and target discourse trees.These criteria are important for long-context and open-ended generation, including document-level machine translation.Source: §6.2.1 · p. 41

6.2.2 Evaluation via Lexical Cohesion

• Lexical cohesion evaluation uses lexical similarity, embedding similarity, entity coreference, and related token-level measures to assess coherence across generated text.A higher entity-coreference score indicates greater coherence, and lexical-cohesion features have been used to improve correlations between BLEU, TER, and human judgments.Source: §6.2.2 · pp. 41, 44

6.2.3 Evaluation via Writing Style

• Writing-style evaluation measures whether generated text maintains stylistic consistency with its context, including part-of-speech distributions and pretrained style-classification scores.Style transfer metrics assess whether a generated sentence exhibits the same style as its surrounding context.Source: §6.2.3 · p. 44

6.2.4 Evaluation with Multiple References

• Multiple references can improve automatic metrics’ correlation with human judgments because long generated text may be valid, fluent, and informative without lexically overlapping a single reference or prompt.This solution has been demonstrated for adequacy and fluency evaluation in machine translation and other NLG tasks.Source: §6.2.4 · p. 44

Evaluation framework

• NLG evaluation comprises human-centric methods, untrained automatic metrics, and machine-learned metrics for increasingly challenging generation systems.Human evaluation remains central, while automatic methods support system monitoring and learned metrics address incomplete references.Source: §7 · p. 45

Human evaluation

• Human evaluation is the gold standard but is expensive, difficult to reproduce, and often lacks consensus and sufficiently detailed reporting.The paper recommends careful study design and reporting of evaluators, sample sizes, procedures, and collected data to improve replicability and comparison.Source: §7 · pp. 45–47

Automatic metrics

• Untrained automatic metrics monitor development, but multiple metrics are often needed because a good metric should correlate with human judgments across quality dimensions.Shared tasks and evaluation platforms could improve consistency and direct comparison between systems.Source: §7 · pp. 45, 47

Machine-learned metrics

• Machine-learned metrics can mimic human judgments when references are incomplete, but may suffer from overfitting and metric gaming (Gao et al., 2019).These risks make learned evaluation models useful but potentially vulnerable to optimization against the metric rather than genuine quality.Source: §7 · p. 45

Factuality and bias

• Evaluation must detect machine-generated text, fake news, factual inconsistency, hallucination, and training-data bias as language models become harder to distinguish from humans.Relevant work includes GLTR, fact verification, factually consistent generation, summary-document consistency checks, and controllable generation, while future systems should measure and prevent nonfactual outputs and reproduced biases.Source: §7 · p. 45

Explainable evaluation

• Future evaluation tools should be easy to use, robust, and explainable so developers can identify causes of bias, repetition, and factual inconsistency.Human-understandable justifications can also increase user trust (Ehsan et al., 2019; Ribeiro et al., 2016, 2018; Thorne et al., 2019).Source: §7 · p. 46

Corpus quality

• High-quality, accurate, timely, complete, clean, relevant, and unbiased corpora with multiple references are essential for reliable evaluation and metrics aligned with human judgments.Missing or group-biased information can prevent models from learning accurate insights and harm downstream task performance (McGuffie & Newhouse, 2020; Gehrmann et al., 2021).Source: §7 · p. 46

Standardization and ethics

• The field needs standardized metrics, human-evaluation criteria, and systematic ethical assessments of offensive or improper outputs, supported by closer collaboration and shared tasks.Ethical evaluation must evolve with neural text-generation advances because even deterministic policies can produce contextually offensive behavior (Zhou et al., 2020).Source: §7 · pp. 46–47