PaperlayerView exact revision on arXiv

Source-linked AI summary

FineSurE: Fine-grained Summarization Evaluation using LLMs

Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, Saab Mansour

arXiv:2407.00908v3cs.CLcs.AI

TL;DR

Automated summary evaluation is needed because human assessment is costly and existing approaches provide limited or coarse-grained evaluation. FineSurE uses LLMs for fine-grained evaluation across faithfulness, completeness, and conciseness, and the paper reports improved human correlation over state-of-the-art methods.

  • Problem

    Human evaluation is costly, while existing LLM-based methods often assess summaries only with coarse summary-level scores.

    Source: §1 · pp. 1–2

  • Method

    FineSurE uses LLMs for sentence-level fact checking and keyfact alignment to assess faithfulness, completeness, and conciseness.

    Source: §1 · p. 2

  • Results

    FineSurE improves human correlation over state-of-the-art automated methods, according to the paper's benchmark comparison.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments indicate FineSurE's potential as an automated text-summarization evaluator.

    Source: §5 · p. 9

    Evaluation is primarily tested on the news domain because fine-grained human-annotation benchmark datasets are limited.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and time-consuming nature of human evaluation. Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likert-scale scores. This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations. To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs). It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment. We compare various open-source and proprietary LLMs as backbones for FineSurE. In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions. The code is available at https://github.com/DISL-Lab/FineSurE-ACL24.

1 Introduction

Automated summarization evaluation remains difficult: conventional metrics correlate weakly with human judgments, while human evaluation is costly. FineSurE addresses the limits of coarse, ambiguously defined LLM evaluation with sentence- and keyfact-level assessment across faithfulness, completeness, and conciseness.

1 Introduction

  • Human evaluation remains essential for accurate assessment despite its cost and time demands.Source: §1 · p. 1
  • Summary-level Likert scores omit sentence-level error counts and mistake types, while coherence and relevance can lack clear definitions.The paper argues for a more precise framework with clearly defined dimensions.Source: §1 · pp. 1–2
  • FineSurE evaluates summaries through sentence-level fact checking and keyfact alignment, producing fine-grained assessments of faithfulness, completeness, and conciseness.Its outputs support analysis of quality issues at both the sentence and keyfact levels.Source: §1 · p. 2
  • Completeness and conciseness assess whether key information is included and kept dense while excluding irrelevant content.The paper presents them as alternatives to coherence and relevance.Source: §1 · p. 2
  • The paper reports comparing FineSurE with similarity-, NLI-, QA-, and LLM-based evaluators, with improved human correlation over state-of-the-art methods.Source: §1 · p. 2

2 Related Work

Prior automated evaluators use similarity, NLI, QA, or LLM-based approaches, but the paper identifies gaps in human correlation, multidimensional coverage, and fine-grained assessment. FineSurE is introduced as a framework for more detailed LLM-based evaluation.

2 Related Work

  • Similarity-based evaluators typically produce a single-dimensional text-similarity score and lack human correlation and multidimensional quality assessment.Source: §2 · p. 3
  • NLI-based evaluators improve faithfulness assessment with finer-grained methods, but focus only on faithfulness.SummaC segments the input into sentences and aggregates scores across sentence pairs.Source: §2 · p. 3
  • QA-based evaluators assess factual consistency, and UniEval extends QA to multiple dimensions, but these methods generally require question- and answer-generation models.Source: §2 · p. 3
  • LLM evaluators show promise but generally lack fine-grained assessment and often focus on faithfulness without other summary-quality dimensions.Source: §2 · p. 3
  • FineSurE introduces an LLM-based framework covering sentence-level fact checking, error-type classification, and keyfact-level alignment.Source: §2 · p. 3

3 FineSurE Framework

FineSurE evaluates faithfulness, completeness, and conciseness using LLM-based fact checking and keyfact alignment. The framework specifies distinct procedures, scoring, prompt choices, and an automatic keyfact-extraction option when human keyfacts are unavailable.

3.1 Evaluation Dimensions

  • Faithfulness measures whether summary information is supported, completeness whether all keyfacts are included, and conciseness whether output avoids information outside the keyfacts.Faithfulness errors are divided into seven categories: one extrinsic and six intrinsic.Source: §3.1 · p. 3

3.2 Evaluation Pipeline

  • Fact checking classifies each summary sentence into one of nine categories and returns its error type with a concise reason.Keyfact alignment checks whether each keyfact is inferred from the summary and records all matching sentence line numbers.Source: §3.2 · p. 4
  • Completeness and conciseness are calculated from keyfact-to-sentence alignments, while faithfulness is the percentage of summary sentences marked error-free.FineSurE also reports sentence error types and keyfact-to-sentence alignments.Source: §3.2 · p. 4

3.3 Prompt Engineering

  • The recommended prompts use instruction, categorization, and reasoning for faithfulness, but instruction alone for completeness and conciseness.Combining all prompt techniques was not always superior.Source: §3.3 · p. 4

3.4 Keyfact Extraction

  • Human-generated keyfacts are preferred, but an LLM can extract them automatically when human keyfacts are unavailable.The paper describes tailored prompts for automatic extraction.Source: §3.4 · pp. 4–5

4 Evaluation

FineSurE is evaluated against human annotations at sentence, summary, and system levels, using separate datasets for sentence-level faithfulness and keyfact alignment. The evaluation compares its agreement, stability, and LLM-backbone performance across faithfulness, completeness, and conciseness.

4 Evaluation

  • Evaluation uses FRANK for sentence-level factuality error types and REALSumm for human keyfacts and their presence in summaries.The study compares FineSurE with similarity-, NLI-, QA-, and LLM-based metrics using sentence-, summary-, and system-level measurements.Source: §4 · p. 5

4.1.1 Faithfulness

  • FineSurE significantly outperforms similarity-, NLI-, and QA-based evaluators across faithfulness evaluation levels, and aligns with human sentence judgments at 86.4% balanced accuracy.It also localizes factuality error types zero-shot better than FRANK-tuned Bart-Large, but LLMs still make numerous exact-category mistakes.Source: §4.1.1 · p. 6, Table 2 · p. 7

4.1.2 Completeness and Conciseness

  • FineSurE with human keyfacts shows very high agreement with human evaluations for completeness and conciseness, significantly surpassing other evaluators.With machine-generated keyfacts, it also maintains higher agreement than other automated evaluators.Source: §4.1.2 · p. 6, Table 3 · p. 7

4.1.3 Stability in Evaluation Results

  • FineSurE maintains much higher summary-level agreement across three runs than G-Eval, with percentage scores derived from sentence- and keyfact-level assessments.Table 4 reports inter-annotator agreement across the three evaluations.Source: §4.1.3 · p. 7, Table 4 · p. 7

4.2 LLMs as Evaluators Comparison

  • Agreement with human scores is generally higher for proprietary LLMs; among open-source models it rises with model size, and instruction tuning improves Mixtral-8x7b.Agreement comparisons use successful prompt-following cases; more recent proprietary models show higher agreement than their predecessors.Source: §4.2 · p. 7, Table 5 · p. 8, Table 6 · p. 8

4.3 Evaluation using FineSurE

  • Across 100 CNNDM examples, model rankings vary by dimension: GPT-3.5-turbo leads faithfulness, GPT-4-turbo completeness, and GPT-4-omni conciseness.Llama3-70B-Inst has comparable performance to state-of-the-art proprietary models; GPT-4-omni’s faithfulness and completeness scores are below GPT-3.5-turbo’s.Source: §4.3 · pp. 8–9, Figure 2 · p. 9

4.2 LLMs as Evaluators Comparison

  • GPT-4-omni improves mean error-localization accuracy by 10% over GPT-4-turbo, while Llama3-70B-Inst averages 49.4%, 7.2% above GPT-4-turbo.Open-source models generally trail proprietary models on error categorization, but Llama3-70B-Inst is an exception.Source: §4.2 · p. 8, Table 7 · p. 9

5 Conclusion

FineSurE evaluates summaries at fine granularity across multiple dimensions. Its experiments compare it with existing evaluators and explore different LLM backbones, with promising results for automated evaluation.

5 Conclusion

  • FineSurE breaks evaluation into fact checking and keyfact alignment, using human-provided or LLM-generated keyfacts to assess summaries at fine granularity.The experiments compare it with existing evaluators, test eight open-source or proprietary LLMs, and assess recent LLM-generated summaries.Source: §5 · p. 9

Limitations

FineSurE is primarily tested on news because fine-grained human-annotation benchmarks are limited; broader domains and additional evaluation aspects remain future work.

Limitations

  • Testing is primarily in the news domain; broader benchmark diversity, domain-specific prompt tuning, and aspects such as toxicity and social bias remain future challenges.The authors cite limited availability of fine-grained human annotations as the reason for the news-domain focus.Source: Limitations · p. 9

Ethics Statement

The authors do not anticipate negative ethical or social impact because the paper focuses on designing an LLM-based automatic evaluator for summarization.

Ethics Statement

  • The authors do not anticipate negative ethical or social impact from this work, which focuses on an LLM-based automatic summarizer evaluator.Source: Ethics Statement · p. 9

A Factuality Error Type

This appendix describes the factuality-error categories, prompts, keyfact extraction, and evaluation measurements used by FineSurE.

A Factuality Error Type

  • Factuality errors are organized into seven categories: out of context, predicate, entity, circumstance, coreference, discourse link, and grammatical.Source: A Factuality Error Type · p. 12

B Main Prompt

  • LLMs perform fact checking and keyfact alignment through two prompts tailored to those tasks.The prompts are shown in Figures 3–4.Source: B Main Prompt · p. 12

C Keyfact Extraction

D Measurement

  • Sentence-level faithfulness uses balanced accuracy, summary-level dimensions use Pearson or Spearman correlation, and system-level scores are ranked and compared with human rankings.The appendix also provides a qualitative example of completeness evaluation using FineSurE and G-Eval.Source: D Measurement · p. 13, Table 9 · p. 14

E Qualitative Analysis for Completeness

A completeness example contrasts G-Eval’s Likert score with FineSurE’s keyfact-based assessment, which more closely matches human judgment.

F Fair Comparison with G-Eval

The fair comparison tunes G-Eval for completeness and conciseness, with and without human keyfacts. FineSurE still outperforms both variants.

G Prompt Engineering

Prompt experiments identify different effective designs for faithfulness versus completeness and conciseness. The appendix also describes summary generation, parsing behavior, and a REALSumm label extension.

G Prompt Engineering

H Summarization using LLMs

I Extension of REALSumm

  • For REALSumm, three annotators label whether each summary sentence implies at least one listed keyfact, extending the data for conciseness evaluation.Source: I Extension of REALSumm · p. 17

Related papers

Loading 2407.00908v3…