Source-linked AI summary
Human-like Summarization Evaluation with ChatGPT
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, Xiaojun Wan
TL;DR
Evaluating summaries is difficult because human judgments are costly and existing automatic metrics remain limited. The paper tests ChatGPT as a human-like evaluator using four human evaluation methods across five datasets. ChatGPT completed annotations relatively smoothly and exceeded common automatic metrics on some datasets, while its performance depended strongly on prompt design.
Problem
Human evaluation of summarization is expensive and time-consuming, while existing automatic metrics remain unsatisfactory in performance, usability, interpretability, and factuality assessment.
Method
The study evaluates ChatGPT with Likert scoring, pairwise comparison, Pyramid, and binary factuality evaluation across five summarization datasets.
Results
ChatGPT completed annotations relatively smoothly and outperformed commonly used automatic evaluation metrics on some datasets.
Takeaways & Limitations
ChatGPT can perform summarization evaluation using varied human protocols, sometimes correlating with human judgments better than existing evaluation metrics.
Takeaways & Limitations
ChatGPT sometimes produces invalid responses, although their fraction is at most about 1%, and the reasons require further exploration.
Abstract
from arXiv · showhide
Evaluating text summarization is a challenging problem, and existing evaluation metrics are far from satisfactory. In this study, we explored ChatGPT's ability to perform human-like summarization evaluation using four human evaluation methods on five datasets. We found that ChatGPT was able to complete annotations relatively smoothly using Likert scale scoring, pairwise comparison, Pyramid, and binary factuality evaluation. Additionally, it outperformed commonly used automatic evaluation metrics on some datasets. Furthermore, we discussed the impact of different prompts, compared its performance with that of human evaluation, and analyzed the generated explanations and invalid responses.
1 Introduction
Summarization evaluation remains difficult because human assessment is costly and existing automatic metrics have limitations in surface matching, factuality assessment, usability, and interpretability. The paper tests whether ChatGPT can conduct human-like evaluation across established human protocols and reports stronger performance than common automatic metrics on some datasets.
- Motivation: Human evaluation is expensive and time-consuming, while automatic metrics can miss summary quality and factual accuracy.ROUGE-style metrics rely on reference-based n-gram overlap and do not use the source document for factuality assessment.
- Motivation: Advanced metrics such as BERTScore, BARTScore, FactCC, and FEQA still have unsatisfactory performance, usability, or interpretability.These methods improve correlation or assess factual consistency, but the paper identifies remaining shortcomings.
- Human-like evaluation: LLMs enable human-like automatic evaluation by using examples and instructions to produce scores, comparisons, labels, and explanations.The approach is flexible across evaluation forms rather than restricted to one numerical metric.
- Study design: ChatGPT was evaluated with Likert scoring, pairwise comparison, Pyramid, and binary factuality evaluation.These are four commonly used human evaluation methods for summarization.
- Findings: ChatGPT completed annotations relatively smoothly and outperformed commonly used automatic metrics on some datasets.The study also examined prompt effects, agreement with human evaluation, explanation quality, and invalid responses.
2 Preliminary
The paper reviews automatic metrics and human protocols used to evaluate summarization. Automatic methods measure textual or semantic similarity and factuality, while human methods assess quality, content units, or source faithfulness across five datasets.
- 2.1 Automatic Evaluation Metrics: ROUGE measures similarity through unigram, bigram, and longest-common-sequence overlap and remains the dominant automatic summarization metric.The review identifies ROUGE-1, ROUGE-2, and ROUGE-L as the most commonly used measures.
- 2.1 Automatic Evaluation Metrics: BERTScore and MoverScore measure semantic similarity using contextual embeddings at the token and n-gram levels.BERTScore uses soft token overlap, whereas MoverScore computes semantic distance from pooled n-gram embeddings.
- 2.1 Automatic Evaluation Metrics: BARTScore treats evaluation as text generation and scores how likely BART is to generate one text from another.It can be applied flexibly from different evaluation perspectives.
- 2.1 Automatic Evaluation Metrics: FactCC and DAE classify summary sentences as factually correct or incorrect, then compute factuality as the proportion judged correct.The summaries are split into individual sentences before classification.
- 2.2 Human Evaluation Methods: Likert scoring rates individual summaries on dimensions such as factual consistency, informativeness, and fluency, usually on a 1-to-5 scale.The study applies it to SummEval and Newsroom; pairwise comparison selects the higher-quality summary from two candidates using TLDR.
- 2.2 Human Evaluation Methods: Pyramid scores the proportion of reference-derived semantic content units present in a generated summary, using REALSumm.Annotators judge whether each extracted SCU appears in the generated summary.
- 2.2 Human Evaluation Methods: Binary factuality evaluation asks whether each generated-summary sentence is faithful to its source document, using QAGS.This protocol directly targets factual correctness rather than general summary quality.
3 Experiments
The experiments used ChatGPT prompts designed to mirror human-evaluation instructions across four summarization evaluation methods, then measured annotation validity and agreement with human judgments. Results were assessed through correlations for Likert scoring and accuracy for the other methods, with stronger performance on some datasets.
- Experimental Setup: The experiments used the ChatGPT API with temperature set to 0 and max_tokens set to 256 to reduce randomness.Other parameters retained their default values.
- Prompt Design: Prompts were designed to be as identical as possible to the original instructions used in human evaluations.
- Prompt Design: Likert scoring asked ChatGPT to rate four dimensions, while Pyramid prompts tested whether semantic content units could be inferred from summaries.The evaluated dimensions differed between SummEval and Newsroom, and Pyramid used up to 16 semantic content units.
- Prompt Design: Binary factuality evaluation presented sentences from generated summaries for factuality judgments, with the method's template shown separately.
- Evaluation: Invalid responses were marked as NAN, while Likert scoring used correlations and the other methods used accuracy against human annotations.The analysis included sample-level, system-level, and dataset-level correlations for Likert scoring.
- Results: On SummEval, ChatGPT performed substantially better than existing metrics for Likert scoring, while on Newsroom it ranked behind only BARTScore_s_h and BARTScore_cnn_s_h.For pairwise comparison, Pyramid, and binary factuality evaluation, annotations were completed relatively smoothly, but a large advantage appeared only on QAGS_XSUM.
4 Analysis and Discussion
The analysis examines prompt effects, agreement with human evaluation, reproducibility, cost, generated explanations, and invalid responses in ChatGPT-based summarization evaluation. Prompt changes substantially affect correlations, while the approach remains cheaper and more controllable but still has accuracy and explanation-quality limitations.
- Impact of different prompts: Prompt changes significantly affect ChatGPT’s human-like evaluation performance, especially system-level correlations.Adding dimension definitions produced modest improvements in a few cases, but no prompt setting consistently achieved higher correlations with human judgments.
- Comparison with human evaluation: ChatGPT remains less aligned with human experts overall, although its fluency system-level correlation reaches 0.889 versus 0.843 for one expert.Human-expert agreement with the average expert score is substantially better than ChatGPT in most cases.
- Variance and reproducibility: ChatGPT evaluations are more reproducible because decoding randomness can be controlled, whereas different human annotators do not assign identical scores.The analysis contrasts controllable model evaluation with the documented difficulty of reproducing manual evaluation.
- Cost: Human-like evaluation using ChatGPT costs about 10 to 20 times less than human evaluation on SummEval.The reported SummEval estimates are about 3.2 USD for ChatGPT versus 60 USD for one annotator.
- Quality of generated explanations: ChatGPT explanations generally agree with its scores but are not necessarily correct, and dimension definitions reduce fluency–coherence correlation from 0.960 to 0.843.Providing explanations alongside scores can help assess whether ChatGPT follows evaluation requirements, but explanation correctness requires further testing.
- Invalid responses: Invalid responses occur in at most about 1% of cases, including refusals, incorrect evaluation formats, and attempts to continue or rewrite summaries.The causes of these invalid responses remain to be explored.
5 Related Work
Concurrent studies also investigate large language models as evaluators of generated language. Their applications include translation quality, general NLG meta-evaluation, ranking, factual consistency, and chain-of-thought assessment.
- Kocmi and Federmann report that LLMs are currently the most advanced evaluators of translation quality.
- Wang et al. test ChatGPT’s evaluator ability on three NLG meta-evaluation datasets.
- Other concurrent work studies ChatGPT for ranking generated content, factual-consistency evaluation in summarization, and chain-of-thought assessment of NLG outputs.
6 Conclusion
ChatGPT can evaluate summaries using multiple human evaluation methods, with higher correlation to human judgments than existing metrics in some instances. Its performance depends strongly on prompt design, while its evaluation is reported as more cost-effective and reproducible than human evaluation.
- ChatGPT can evaluate summaries using various human evaluation methods and sometimes correlates with human judgments better than existing metrics.
- ChatGPT's summarization-evaluation performance is highly dependent on prompt design.
- Human-like evaluation with ChatGPT is more cost-effective and reproducible than human evaluation.
- ChatGPT's generated explanations are consistent with its scoring.