PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Text Generation: A Survey

Asli Celikyilmaz, Elizabeth Clark, Jianfeng Gao

arXiv:2006.14799v2cs.CLcs.LG

TL;DR

NLG evaluation is difficult because open-ended tasks permit multiple plausible outputs, while human judgments are costly and some quality dimensions are difficult to assess consistently. The paper surveys human-centric, untrained automatic, and machine-learned metrics, emphasizing neural systems and task-specific evaluation. It concludes by highlighting the need for improved corpus quality and closer standardization through challenges or shared tasks.

  • Problem

    Open-ended NLG outputs are difficult to evaluate reliably: human evaluation is costly, and some metrics do not align well with human judgments across tasks.

  • Method

    The paper comprehensively surveys NLG evaluation methods in three categories and examines their application to summarization and long-text generation.

  • Results

    The survey identifies task-specific evaluation as requiring multiple metrics and reviews evaluation challenges across neural NLG systems.

  • Takeaways & Limitations

    Improving NLG evaluation requires higher-quality corpora with multiple references and closer standardization through shared challenges or tasks.

  • Takeaways & Limitations

    Human evaluation remains expensive and time-consuming, can require domain expertise, and may be inconsistent or poorly suited to dimensions such as diversity.

Abstract

from arXiv · show

The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two examples for task-specific NLG evaluations for automatic text summarization and long text generation, and conclude the paper by proposing future research directions.

1. Introduction

Natural language generation spans many tasks that produce coherent text from varied inputs, but evaluating open-ended outputs is difficult because multiple responses can be valid. This survey organizes NLG evaluation methods, especially for neural systems, and examines their use across tasks and applications.

  • 1. Introduction: NLG systems transform inputs such as tables, prompts, or images into coherent and understandable text across many language tasks.Applications include chatbots, summarization, translation, question generation, and long-text generation.
  • 1. Introduction: Open-ended generation makes evaluation challenging because different outputs may all be plausible for the same input.Human evaluation remains the gold standard, but its expense motivates automatic metrics for progress tracking and system optimization.
  • 1. Introduction: The survey groups NLG evaluation into human-centric metrics, untrained automatic metrics, and machine-learned metrics.It discusses progress, challenges, and future directions for each category.
  • 1. Introduction: Human-centric evaluation uses people to rate or compare generated texts, apply task-specific criteria, assess general qualities, or distinguish machine from human text.Criteria may include entity correctness, fluency, and grammar.
  • 1. Introduction: Untrained automatic metrics compare generated and reference texts using string or content overlap, distance, lexical diversity, or distributional similarity.The metric should match the intended design goals of the NLG system.
  • 1. Introduction: The paper applies its survey framework to shared tasks, automatic document summarization, and long-text generation before discussing future research directions.It focuses on recently developed metrics for neural text generation systems.

2. Human-Centric Evaluation Methods

Human judgments remain central because they provide the best insight into generated-text quality, but they are costly, difficult to reproduce, and vary in suitability across evaluation dimensions. The section describes intrinsic and extrinsic evaluations, comparative and individual judgments, common quality dimensions, evaluator recruitment, and agreement challenges.

  • Motivation: Human evaluation is the gold standard for NLG because automatic metrics still fall short of replicating human decisions.Human judgments are also used to validate automatic metrics through correlation with human evaluations.
  • Challenges: Human evaluations are expensive, time-consuming, inconsistent, and sometimes poorly suited to dimensions such as diversity, limiting reproducibility and cross-system comparison.The survey also notes that agreement is often low, although maximizing inter-annotator agreement may be inappropriate for rich and varied natural language.
  • Intrinsic evaluation: Intrinsic evaluations ask people to judge generated texts overall or along dimensions such as fluency, coherence, and correctness.Evaluators may score samples individually or compare and rank outputs.
  • Intrinsic evaluation: Individual scoring is simple but can be inconsistent, whereas comparative judgments may improve inter-annotator agreement without measuring absolute text quality.Comparative methods can require many head-to-head comparisons or complex rankings when many models are involved.
  • Evaluation dimensions: Human evaluation commonly considers factuality, fluency, adequacy, grammaticality, style, formality, tone, and typicality, but task-specific dimensions are not consistently established or reported.Over 70% of generated single-sentence summaries contained hallucinations in one cited study, while about 25% of generation papers failed to mention evaluation dimensions.
  • Extrinsic evaluation: Extrinsic evaluation measures downstream-task success and is considered more meaningful, but its expense and difficulty make intrinsic evaluation more common.Crowdsourcing enables large-scale, time-efficient, and relatively inexpensive evaluation, but raises quality-control, demographic, and user-alignment concerns.

3. Untrained Automatic Evaluation Metrics

Untrained automatic metrics provide fast, efficient alternatives to costly human evaluation, but their usefulness depends on selecting metrics suited to each NLG task and human judgment criteria.

  • Human evaluation remains important because NLG outputs are often open-ended, allowing multiple plausible responses or summaries.
  • Researchers use automatic metrics because human evaluation is costly, time-consuming, and not always repeatable.
  • Untrained metrics compare generated text with human-written references using string overlap, distance, content overlap, or lexical diversity.
  • These metrics are fast and efficient for tracking day-to-day development, including comparisons among models trained with different hyperparameters.
  • Limited human studies report weaker correlations for most untrained metrics, with correlation strength depending on the human evaluator.

3.1 n-gram Overlap Metrics for Content Selection

N-gram overlap metrics compare generated and reference texts, but their relationship with human judgments varies across tasks and can miss semantic, structural, or global coherence.

  • N-gram overlap metrics measure matching between machine-generated and human-authored texts, commonly using precision, recall, and F-scores.
  • Recent findings report that BLEU may correlate poorly with human judgments outside machine translation and can yield grammatically correct but semantically incoherent text.
  • BLEU has been used for summarization, image captioning, conversation, and language generation beyond machine translation.
  • BLEU is fast and enables model comparisons on the same task, but it does not represent semantic meaning or sentence structure.
  • ROUGE emphasizes recall and is more interpretable than BLEU, but n-gram matching omits narrative flow, grammar, topical flow, and factual correctness.
  • METEOR extends exact matching with stemming and synonym matching, addressing reference-translation variability and correlating well with sentence- or segment-level judgments.

3.2 Distance-Based Evaluation Metrics for Content Selection

Distance-based metrics represent text units as vectors and measure similarity through distances, ranging from edit operations to word- and sentence-embedding transport.

  • Distance-based metrics represent two text units as vectors and interpret smaller distances as greater similarity.
  • Word error rate counts the percentage of insertions, deletions, and replacements needed to transform a hypothesis into a reference.
  • WER is lower-bounded by zero but unbounded above, making absolute evaluation difficult, and it can correlate weakly with human evaluation.
  • Embedding-based metrics use pretrained representations so semantically similar tokens can have similar vector representations without training to mimic human judgments.
  • WMD combines bag-of-words frequencies with word-embedding similarity and is hyperparameter-free, interpretable, and associated with high retrieval accuracy.
  • WMD works well for short texts but becomes prohibitively costly for long documents and loses sentence relations through its bag-of-words representation.
  • SMD moves weighted sentence embeddings between documents, and on summarization it correlated better with human judgments than ROUGE.

3.3 n-gram-Based Diversity Metrics

Diversity metrics measure lexical variety and repetition, but diversity should be interpreted alongside text quality because high variety can accompany weaker outputs.

  • Lexical diversity measures the breadth and variety of word usage and is relevant to conversational bots, stories, question generation, and abstractive question answering.
  • Type-token ratio divides unique words by total words to estimate vocabulary richness.
  • Type-token ratio decreases as documents lengthen because new tokens become less likely, motivating length-adjusted measures such as HD-D.
  • Self-BLEU averages BLEU scores computed for each generated sentence against the others as references; lower Self-BLEU indicates higher diversity.
  • Reported weaknesses include Self-BLEU’s inconsistent ability to generate diverse output or detect mode collapse in GAN-based text generation.

3.4 Explicit Semantic Content Match Metrics

Explicit semantic content metrics compare generated and reference texts through meaning-bearing units rather than surface word overlap. They include manually annotated content units, scene graphs, sentence similarity, paraphrase, entailment, and related semantic comparisons.

  • Semantic content matching metrics extract explicit semantic information beyond n-grams and generally correlate well with human judgments.
  • Pyramid: The pyramid method compares model summaries with human references using annotated summarization content units.Its automated variant, PEAK, assigns pyramid weights automatically and correlates well with human judgments.
  • SPICE: SPICE parses reference and hypothesis captions into scene graphs of objects, attributes, and relationships, then compares their semantic propositions.SPICE has shown strong correlation with human ratings and captures broader semantic similarity than n-gram matching.
  • Composite semantic metrics: SPIDER combines SPICE’s semantic graph similarity with CIDER’s n-gram similarity because SPICE alone can produce wordy, repetitive captions.
  • Sentence-level semantic methods: Sentence-level semantic evaluation includes machine comprehension, semantic textual similarity, paraphrase identification, and textual entailment across generation tasks.STS scores use weighted Pearson correlation between predicted and human-annotated similarity, with higher scores indicating better prediction.

3.5 Syntactic Similarity-Based Metrics

Syntactic similarity metrics evaluate generated text against references at the structural level, using part-of-speech information and syntactic analyses such as dependency trees.

  • Syntactic similarity metrics compare reference and hypothesis texts at a structural level, capturing grammatical or sentence-structure similarity.
  • POS-based metrics: POS-based metrics use part-of-speech tags, bilingual phrase-table synonyms, POS n-grams, morphemes, or lexical probabilities to compare texts.POS information has been applied to machine translation, story generation, summarization, and question generation.
  • Syntactic analysis: Syntactic analysis can enrich generation evaluation by representing grammatical structure with dependency trees.

4. Machine-Learned Evaluation Metrics

Machine-learned evaluation metrics address weaknesses of overlap-based and embedding-based metrics, especially for diverse or open-ended generation. They learn semantic, regression, human-judgment, factuality, or composite quality signals for tasks including dialogue, summarization, and captioning.

  • Overlap-based metrics fail when plausible outputs share few words with references, while embedding metrics remain limited when outputs are semantically different.Machine-learned evaluators can instead learn quality dimensions such as factual correctness, naturalness, and fluency from human judgments.
  • Sentence semantic similarity: Sentence representation methods embed generated and reference sentences for semantic comparison, with Skip-Thoughts learning representations by predicting contextual sentences.
  • Sentence semantic similarity: Transformer-based contextual encoders such as BERT and RoBERTa outperform earlier representations but are more computationally expensive and require task-specific fine-tuning.Sentence-BERT was proposed to make cosine similarity between sentence embeddings more practical.
  • Regression-based evaluation: RUSE treats evaluation as regression, encoding hypothesis and reference translations and learning a scalar quality score from their matched representations.
  • Open-ended generation: Human evaluation remains difficult for diversity and creativity because judges may lack knowledge of the model’s training dataset.
  • Evaluation with human judgments: ADEM learns from human judgments to score dialogue responses, correlates well with human ratings, but is conservative and scores plausible responses lower.It is trained end-to-end with squared error loss and L-2 regularization.
  • Factual correctness: Generated text can contain up to 30% factual inconsistencies, motivating evaluation methods that verify whether outputs adhere to source facts.
  • Factual correctness: Factuality evaluators use question-answering over source-linked questions, while PARENT aligns table entities and measures overlap, entailment, and related signals.PARENT correlates better with human judgments than several n-gram and information-extraction metrics in a large-scale study.

5. Shared Tasks for NLG Evaluation

Shared tasks benchmark NLG systems using common data and evaluation metrics, supporting comparison while addressing evaluation gaps across emerging generation tasks.

  • Shared tasks use the same data and evaluation metrics to benchmark NLG models efficiently.
  • NLG shared tasks address the lack of established evaluation pipelines for many generation tasks.
  • The GRE Challenge evaluated referring-expression content determination, including minimality relative to an original definition.
  • GIVE evaluated instruction generation by having systems guide human players through tasks in simulated 3D environments.

5.4 Regular Expression Generation from Attribute Sets

This section describes shared-task examples spanning referring-expression generation, semantic text generation, and RDF-to-text generation.

  • Regular Expression Generation from Attribute Sets: The task asks how attribute sets can produce semantic content that uniquely identifies an intended entity among distractors.
  • SemEval Task 9 focused on text generation from Abstract Meaning Representation.
  • WebNLG benchmarked generation from RDF triples across microplanning operations such as referring expressions, aggregation, lexicalization, and realization.

5.7 E2E NLG Challenge

The E2E NLG Challenge supplied data for response-generation evaluation, while later efforts expanded benchmarking through broader datasets, metrics, and testing environments.

  • E2E NLG Challenge: The 2018 E2E NLG Challenge provided a high-quality, large training dataset for evaluating spoken-dialog response generation.
  • E2E NLG Challenge: E2E required models to jointly learn sentence planning and surface realization without costly meaning-to-reference alignment.
  • Earlier data-to-text work largely introduced datasets and benchmarks rather than organizing challenges.
  • Introduced in ACL 2021, GEM aimed to measure NLG progress while adding datasets, evaluation metrics, and human evaluation standards.
  • GEM provides an environment for testing different NLG tasks and evaluation strategies.

6. Examples of Task-Specific NLG Evaluation

Because no single metric fully captures NLG quality, the paper illustrates joint metric use for automatic summarization and long-text generation.

  • NLG metrics evolve with advances in neural architectures and increasing computational resources.
  • Defining an accurate, trustworthy, or efficient evaluation metric remains difficult.
  • The paper presents automatic document summarization and long-text generation as tasks requiring multiple metrics to assess different quality aspects.
  • Table 4 lists the metrics used in each example text-generation project.

6.1 Automatic Document Summarization Evaluation

Automatic summarization evaluation measures generated summaries against reference-free criteria, source documents, or reference summaries, using intrinsic and extrinsic perspectives. Key dimensions include content, quality, factual consistency, and downstream usefulness.

  • Summarization evaluation uses reference-independent criteria, source-document closeness, or reference-summary closeness, organized into intrinsic and extrinsic categories.
  • Intrinsic evaluation: Content evaluation commonly uses ROUGE to compare generated and reference summaries, but its correlation with human judgments is lower for topic-focused meeting summarization.
  • Intrinsic evaluation: Quality evaluation remains challenging because no established definition of a good-quality summary determines which metrics are most suitable.
  • Intrinsic evaluation: Human quality criteria include coherence and cohesion, readability and fluency, focus, and informativeness.
  • Intrinsic evaluation: Factual consistency is an important evaluation target because summarization models frequently generate claims that conflict with source documents.
  • Extrinsic evaluation: Extrinsic metrics assess how generated summaries affect downstream tasks such as relevance assessment, reading comprehension, and question answering.

6.2 Long Text Generation Evaluation

Long-text generation evaluation must assess multi-sentence or multi-paragraph outputs whose coherence and quality extend across sentences and discourse structure. Automatic evaluation therefore examines lexical, stylistic, semantic, and reference-diversity-related properties.

  • Long-text generation covers multi-sentence outputs such as paragraphs and documents, but statistical neural models have been insufficient for generating coherent long text.
  • Evaluation requires criteria for inter-sentence and interparagraph coherence in language style and semantics, with this section focusing on automatic methods.
  • Discourse and coherence: Discourse structure is crucial because longer contexts contain sections whose topical flow and organization must be evaluated.
  • Lexical cohesion: Lexical cohesion metrics assess how textual units are linked, including lexical similarity measures commonly used in story generation.
  • Lexical cohesion: Entity co-reference measures the proportion of generated-sentence entities linked to entities in context, with higher scores indicating higher coherence.
  • Style: Style evaluation can measure consistency between context and continuation using part-of-speech category distributions or pretrained style-classification scores.
  • Multiple references: Long generated text may be fluent, valid, and informative without lexical overlap with a reference or prompt, making multiple references an effective way to improve metric–human correlation.

7. Conclusions and Future Directions

The paper concludes that NLG evaluation must become more standardized, reproducible, explainable, and capable of addressing ethical risks as generation systems and tasks grow more challenging. It recommends better corpora, human-evaluation practices, shared evaluation efforts, and tools that expose reasons for scores.

  • Human evaluation remains the gold standard, but it is expensive to execute and difficult to reproduce.
  • Untrained automatic metrics are widely used to monitor system-development progress, while multiple metrics can assess different quality aspects.
  • Machine-learned metrics can mimic human judges when references are incomplete, but may suffer from overfitting and metric gaming.
  • Future evaluation challenges include detecting machine-generated text and fake news, especially as language models become harder to distinguish from human-authored text.
  • Explainable evaluation tools could help developers identify quality problems such as bias, repetition, and factual inconsistency, while increasing user trust.
  • Future directions: High-quality datasets with multiple references are essential for reliable evaluation and for developing automatic metrics that correlate with human judgments.
  • Future directions: NLG evaluation lacks standard criteria across task-specific metrics and human-evaluation methods, motivating closer collaboration and shared tasks.
  • Future directions: Researchers should report human-evaluation procedures and collected data clearly to improve replicability and enable direct comparisons across results.
Loading 2006.14799v2…