PaperlayerView exact revision on arXiv

Source-linked AI summary

GPTScore: Evaluate as You Desire

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, Pengfei Liu

arXiv:2302.04166v2cs.CL

TL;DR

Text-generation quality evaluation remains difficult because existing methods provide limited aspect coverage and customization. GPTScore uses generative pre-trained models with natural-language task and aspect instructions, leveraging zero-shot instruction and optional in-context demonstrations. Experiments report improved evaluation reliability and flexibility across multiple settings, while comprehensive prompt-engineering exploration remains future work.

  • Problem

    Existing text evaluation methods cover limited aspects, inadequately address multi-aspect definitions, and commonly require supervised signals or annotated samples.

  • Method

    GPTScore scores generated text with generative pre-trained models using task and aspect instructions, conditional generation probabilities, and optional exemplar demonstrations.

  • Results

    Experiments across natural language generation tasks found that instructions improve evaluation reliability and demonstrations can further enhance performance across evaluation settings.

  • Takeaways & Limitations

    GPTScore supports customized, multi-aspect, training-free text evaluation through natural-language descriptions and generative pre-trained models.

  • Takeaways & Limitations

    The paper leaves a more comprehensive exploration of prompt engineering for future work.

Abstract

from arXiv · show

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation--how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.

1. Introduction

Text-generation quality evaluation lags behind generation, with existing methods limited in aspects, difficult to customize, and often dependent on supervised signals. GPTScore uses generative pre-trained models’ zero-shot instruction and in-context learning to support customized, multi-aspect, training-free evaluation.

  • Existing text evaluation methods cover limited aspects, while multi-aspect approaches inadequately define aspects and their relationships.
  • Generative pre-trained models enable multi-aspect, customized, and training-free text evaluation through zero-shot instruction and in-context learning.
  • GPTScore evaluates generated text by measuring its conditional generation probability under a specified evaluation aspect.Higher-quality text for an aspect is expected to be more likely generated from the given context.
  • The framework establishes an evaluation protocol from task specifications and aspect definitions, optionally supplemented with exemplar samples.Each evaluated sample is presented together with this protocol.
  • Experiments across common natural language generation tasks found that task and aspect instructions improve reliability, while demonstrations can enhance evaluation further.
  • GPT3-text-davinci-003 performs worse than GPT3-text-davinci-001 in most evaluation settings, motivating further study of human-feedback instruction learning.

2. Preliminaries

Text evaluation assigns quality scores to hypothesis texts for specified aspects, using human protocols or automated metrics, while meta-evaluation measures metric reliability against human judgments. Evaluation strategies differ in how correlations are aggregated across samples and systems.

  • Text evaluation assesses hypothesis text quality for a specified aspect using manual protocols or automated metrics.
  • The evaluation formulation represents the hypothesis text as h, the aspect as a, and optional additional texts as S.Function f can be a human evaluation process or an automated metric.
  • Meta-evaluation measures automated-metric reliability by correlating automated scores with human judgments using Spearman or Pearson correlation.
  • Sample-level evaluation computes a correlation separately for each sample across multiple system outputs, then averages those correlations.
  • Dataset-level evaluation computes correlation over system outputs from all n samples.
  • The study uses sample-level evaluation for summarization, data-to-text, and machine translation, and dataset-level evaluation for dialogue response generation.

3. GPTSCORE

GPTSCORE uses generative pre-trained language models to evaluate text by conditioning token probabilities on task descriptions, aspect definitions, and context. The framework supports demonstrations and depends on manually designed prompt templates.

  • Generative Pre-trained Language Models: The framework uses generative models that continue prompted text autoregressively, including encoder-decoder and decoder-only backbones.The paper discusses encoder-only, encoder-decoder, and decoder-only model categories, while generation produces a textual continuation from a prompt.
  • GPTSCORE: GPTSCORE scores a hypothesis by the probability assigned to its tokens under an instruction and context.The instruction combines the task description and evaluation aspect, while context may include source or reference text.
  • GPTSCORE: Each token receives a weight in the score, and the implementation treats all token weights equally.The prompt template T defines how task description, aspect definition, and context are organized, usually through task-dependent prompt engineering.
  • Few-shot with Demonstration: Few-shot demonstrations can be added by extending the prompt template with annotated examples.The framework supports demonstrations because generative pre-trained models can perform tasks better when prefixed with a few examples.
  • Choice of Prompt Template: Prompt templates organize task descriptions, aspect definitions, and context, with this paper using official OpenAI prompts for GPT3-based models.Instruction-based models use prompts from NaturalInstruction, while broader prompt-engineering exploration is left for future work.
  • Selection of Scoring Dimension: GPTSCORE variants score a hypothesis from either source text or a gold reference, selected to align with human-judgment protocols.The paper distinguishes src->hypo, p(hypo|src), from ref->hypo, p(hypo|ref).

4. Experimental Settings

The experiments cover four natural language generation tasks using 37 datasets and 22 evaluation aspects. They compare GPTSCORE and established automatic metrics across multiple pre-trained backbones and task-specific scoring variants.

  • Tasks, Datasets, and Aspects: 37 datasets across four generation tasks provide the experimental coverage: dialogue response generation, text summarization, data-to-text, and machine translation.The task suite includes FED, four summarization datasets, BAGEL, SFRES, and MQM-2020 Chinese-to-English.
  • Tasks, Datasets, and Aspects: 22 evaluation aspects are studied across the datasets, with aspect coverage varying by task and dataset.Table 8 summarizes the task, dataset, and aspect assignments, while Table 1 defines the evaluated aspects.
  • Scoring Models: The scoring-model comparison includes ROUGE, PRISM, BERTScore, MoverScore, DynaEval, BARTScore variants, and GPTSCORE.GPTSCORE is instantiated with GPT3, OPT, FLAN-T5, and GPT2, while the baselines include reference-based and dialogue-specific methods.
  • Scoring Dimension: Scoring direction is chosen to match human-judgment protocols, using src->hypo for specified dialogue, summarization, data-to-text, and translation aspects.The paper mainly evaluates English text; source-language differences and non-standard data-to-text formats motivate some src->hypo choices.
  • Experimental Design: The study constructs sub-datasets for meta-evaluation because GPT3 API calls are expensive and selects MQM samples with human scores where possible.For MQM, the selected scored aspects are ACC, MQM, and FLU.

5. Experiment Results

Experiments evaluate GPTScore across summarization, machine translation, data-to-text, and dialogue response generation using vanilla, instruction, and instruction-plus-demonstration settings. Instructions generally improve performance, demonstrations add gains in several tasks, and model structure affects the benefit.

  • Experimental Setup: The study compares vanilla, instruction, and instruction-plus-demonstration evaluation settings across 28 scoring models and multiple generation tasks.Significance testing marks improvements over vanilla and, where applicable, over instruction-only evaluation.
  • Text Summarization: Instruction significantly improves text summarization evaluation, with some small models outperforming BARTScore+CNN on the CON aspect.OPT350m, FT5-small, and FT5-base exceed BARTScore+CNN on CON when instructed.
  • Text Summarization: GPT3-d03 significantly outperforms GPT3-d01 in text summarization, while GPT3-d01 is only barely significantly better than GPT3-c01.The GPT3 variants are compared across four summarization datasets using correlation metrics.
  • Machine Translation: Instruction significantly improves machine-translation evaluation across ACC, FLU, and MQM, while instruction plus demonstration further improves GPT3, GPT2, OPT, and FT5 models.GPT3-c01 achieves performance comparable to GPT3-d01 and GPT3-d03 at lower cost.
  • Data-to-Text: Instruction significantly improves data-to-text evaluation, and demonstrations further improve NAT and FLU, especially for decoder-only GPT2 and OPT models.With instruction and demonstration, GPT3 variants achieve much higher average performance than GPT2, OPT, and FT5 on BAGEL with replaced named entities.
  • Dialogue Response Generation: GPT3-d01 outperforms GPT3-d03 by 40.8 on FED Turn-level and 5.5 on FED dialogue-level average Spearman correlation, despite equal model size.GPT3-based models also generalize more strongly than the evaluated BART-based models in dialogue response generation.

6. Ablation Study

The ablation studies examine how demonstration examples and aspect-definition composition affect GPTScore performance. Demonstrations generally help, but gains can saturate and sparse examples can hurt smaller models.

  • Effectiveness of Demonstration: The study varies demonstration size K across GPT3-based models ranging from 350M to 175B on machine translation.
  • Effectiveness of Demonstration: Demonstrations significantly improve evaluation performance across the examined aspects.
  • Effectiveness of Demonstration: When K>4, ACC performance is difficult to improve further, indicating an upper bound on demonstration gains.
  • Effectiveness of Demonstration: With K=1, small models such as GPT3-a01 can degrade because the examples are one-sided.
  • Aspect Definition Composition: Aspect-correlation analysis uses INT on the FED-Turn dialogue response generation dataset and combines its definition with other aspect definitions.

X Aspect Aspect Definition Sp

The study tests whether combining related aspect definitions improves INT evaluation. GPT3-c01 performance rises as additional correlated aspects are incorporated.

  • Aspect Definition Composition: The INT definition changes from asking whether a response is interesting to asking whether it is interesting, specific, and engaging.
  • Aspect Definition Composition: INT alone scores 30.8, while combining INT with ENG and SPE raises performance to 48.6.
  • Aspect Definition Composition: Combining five aspects—INT, ENG, SPE, COR, and REL—achieves the best reported performance of 51.4.
  • Aspect Definition Composition: Figure 7 ranks aspects by Spearman correlation with INT and tracks INT correlation as its definition is modified.
  • Aspect Definition Composition: The analysis uses GPT3-c01 as the scoring model.

7. Conclusion

GPTScore leverages generative pre-trained models to provide customizable, multi-faceted, training-free text evaluation. The framework supports broad evaluation coverage without a learning process.

  • Conclusion: GPTScore supports 22 evaluation aspects across 37 datasets without any learning process.
  • Conclusion: The framework uses emergent abilities of generative pre-trained models to address intricate and changing evaluation requirements.
  • Conclusion: GPTScore’s stated benefits are customizability, multi-faceted evaluation, and training-free operation.
  • Conclusion: The work studies multiple pre-trained language models, including GPT3 with a model size of 175B.

B. Tasks, Datasets, and Aspects

The experiments cover four natural language generation tasks, multiple datasets, and diverse evaluation aspects. The benchmark includes dialogue, summarization, data-to-text, and machine translation settings.

  • Scope: The study covers 9 datasets and 22 evaluation aspects across its task suite.
  • Tasks: The four tasks are Dialogue Response Generation, Text Summarization, Data-to-Text, and Machine Translation.
  • Dialogue Response Generation: Dialogue Response Generation includes FED data with turn- and dialogue-level human annotations across multiple evaluation aspects.
  • Text Summarization: Text summarization spans semantic coverage, informativeness, relevance, fluency, coherence, and factuality.
  • Data-to-Text: Data-to-Text uses BAGEL and SFRES, covering informativeness, naturalness or relevance, and quality or fluency.
  • Machine Translation: Machine translation uses MQM-2020 for Chinese-to-English evaluation of accuracy, fluency, and MQM scores.
  • Experimental Analysis: The demonstration-size case study uses five GPT3-based models and reports results in Figure 6 and Table 9.

C.2. Partial Order of Evaluation Aspect

The paper investigates combining evaluation aspects to improve performance and defines task-specific instructions across four text generation tasks.

  • Combining different evaluation aspects is investigated for further performance gains, with aspect definitions and Spearman-correlation changes summarized for INT.The scoring model used in the aspect-combination analysis is GPT3-c01.
  • The studied tasks are text summarization, machine translation, data-to-text, and dialogue response generation.Dialogue response generation is converted into a boolean question-answering task, with the aspect definition incorporated into the question.
  • Instructions are designed for different evaluation aspects across summarization, translation, and data-to-text tasks.The instruction design uses source, hypothesis, and reference texts, with task-specific evaluation directions.

E. Experiment Results

The experiments compare nine baseline evaluators with 19 GPTScore models across summarization, translation, and data-to-text datasets. Results are organized by task and dataset, with instruction and demonstration variants reported alongside vanilla evaluators.

  • Experimental setup: Nine baseline models and 19 GPTScore models based on GPT3, GPT2, OPT, and FLAN-T5 are evaluated.The baselines include ROUGE variants, BERTScore, MoverScore, PRISM, and BARTSCORE variants.
  • Experimental setup: The reported experiments cover text summarization, machine translation, and data-to-text datasets including BAGEL and SFRES.The results are presented in Tables 13–16, with summarization, translation, and data-to-text results separated by dataset.
  • Evaluation variants: The tables distinguish vanilla, instruction, and instruction-plus-demonstration evaluators, marking significant improvements with † and ‡.The notation is defined for the summarization, translation, and data-to-text result tables.
  • Text summarization: GPT3-a01 reaches 42.5 Spearman correlation for NEWSROOM IST+DM, 41.7 for QXSUM IST+DM, and 40.2 for COH IST+DM.These values are reported with instruction and demonstration variants in the summarization results.
  • Text summarization: GPT3-c01 reaches 47.5 Spearman correlation for NEWSROOM IST+DM, 39.9 for QXSUM IST+DM, and 44.2 for COH IST+DM.The table marks these instruction-based results as significantly outperforming the corresponding vanilla or instruction variants where indicated.
Loading 2302.04166v2…