PaperlayerView exact revision on arXiv

Source-linked AI summary

Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

Yiming Wang, Zhuosheng Zhang, Rui Wang

arXiv:2305.13412v1cs.CL

TL;DR

Noisy reference summaries in standard news datasets complicate evaluation of summarization systems. The paper creates expert-written element-aware test sets and uses SumCoT to guide LLMs through core-element extraction before summary generation. SumCoT summaries are especially faithful to extracted elements on CNN/DailyMail, with performance depending on model size.

  • Problem

    Standard reference summaries can contain redundancy and factual hallucinations, and have been reported to perform poorly on human-assessment dimensions such as coherence, consistency, and relevance.

    Source: §1 · p. 1

  • Method

    The paper creates expert-written element-aware test sets and uses SumCoT to extract Entity, Date, Event, and Result before guiding LLMs to generate summaries.

    Source: §1 · p. 2

  • Results

    SumCoT summaries are especially faithful to extracted elements on CNN/DailyMail, while BBC XSum coverage is lower and extraction is effective only with larger models.

    Source: How does SumCoT affect summary writing? · p. 8

  • Takeaways & Limitations

    Element-aware test sets are designed to assess diverse language models' general summarization capabilities more thoroughly.

    Source: §6 · p. 9

    The expert test sets are smaller than the original datasets, and their domain balance is not fully considered.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automatic summarization generates concise summaries that contain key ideas of source documents. As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for performance benchmarking. However, the reference summaries of those datasets turn out to be noisy, mainly in terms of factual hallucination and information redundancy. To address this challenge, we first annotate new expert-writing Element-aware test sets following the "Lasswell Communication Model" proposed by Lasswell (1948), allowing reference summaries to focus on more fine-grained news elements objectively and comprehensively. Utilizing the new test sets, we observe the surprising zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic evaluation metrics of LLMs' zero-shot summaries in prior work. Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more fine-grained details of source documents into the final summaries that correlate with the human writing mindset. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively. Dataset and code are publicly available at https://github.com/Alsace08/SumCoT.

1 Introduction

Standard news-reference summaries can be noisy, while expert-written element-aware test sets offer a finer-grained evaluation. On these test sets, zero-shot LLMs are competitive with strong fine-tuned PLMs, and SumCoT guides LLMs through element extraction before summary generation.

1 Introduction

  • Standard reference summaries can contain information redundancy and factual hallucinations and perform poorly on coherence, consistency, and relevance (Kryscinski et al., 2019; Maynez et al., 2020; Stiennon et al., 2020; Fabbri et al., 2021).The paper notes that lexical-overlap evaluation reflects fit to these standard datasets.Source: §1 · p. 1
  • On the element-aware test sets, zero-shot LLMs are highly competitive with some strong fine-tuned PLMs, while PLM performance declines relative to standard test sets.The authors suggest this may help explain why GPT-3 summaries favored by people can score poorly under automatic evaluation on noisy testing domains.Source: §1 · p. 2
  • Three news experts rewrote CNN/DailyMail (Nallapati et al., 2016) and BBC XSum (Narayan et al., 2018) summaries under the Lasswell Communication Model, using a fine-grained 5W1H protocol.The protocol groups who and where as Entity, and why and what as Event.Source: §1 · p. 2
  • SumCoT first prompts LLMs to extract Entity, Date, Event, and Result, then uses their answers to guide generation of summaries with more critical details.The approach aims to produce fine-grained summaries step by step that better correlate with the element-aware writing pattern.Source: §1 · p. 2

2 Element-aware Summary Test Set

The paper constructs expert-written, element-aware test sets for CNN/DailyMail and BBC XSum, using a protocol that combines core news elements with quality requirements. The authors report stronger element coverage and human preference than in dataset-specific summaries.

2.1 Data Construction

  • Three news experts wrote summaries for 200 randomly sampled documents per dataset, with one expert leading and the others reviewing until agreement.The datasets are CNN/DailyMail and BBC XSum; each summary took about 25–30 minutes for CNN/DailyMail and 15–20 minutes for BBC XSum.Source: §2.1 · pp. 2–3

2.2 Writing Protocols

  • The protocol requires faithful Entity, Date, Event, and Result coverage while also specifying fluency, coherence, consistency, and relevance.The macro demands are intended to ensure professionalism and objectivity and to prevent simple stacking of elements.Source: §2.2 · p. 3

2.3 Overall Quality

  • Element-aware summaries have fewer novel n-grams while broadly matching original-summary lengths, and annotators preferred them on logical coherence and factual importance.The authors describe the lower novel-n-gram rate as reflecting greater faithfulness without heavy replication.Source: §2.3 · pp. 3–4, Table 1 · p. 3, Figure 3 · p. 4

2.4 Element-aware Characteristic

  • Element-aware test sets show a significant advantage over dataset-specific sets in element-aware characteristics, especially recall, where standard sets miss many fine-grained details.The comparison evaluates precision, recall, and F1 across all four elements.Source: Table 2 · p. 4, §2.4 · p. 4

3 Preliminary Comparison: Zero-shot LLMs Versus Fine-tuned PLMs

The paper compares zero-shot GPT-3 and fine-tuned PLMs using both standard and expert-written news references, then assesses human preferences. GPT-3 performs competitively on element-aware references, while its scores shift substantially relative to dataset-specific references.

3.2 Main Results

  • In Table 3’s comparison with dataset-specific tests, zero-shot 175B GPT-3 improved on element-aware tests by +7.65/+6.22/+6.74 points for CNN/DailyMail and +11.75/+7.26/+9.56 for BBC XSum.Fine-tuned PLMs performed worse on element-aware than dataset-specific test sets.Source: Table 3 · p. 5, §3.2 · pp. 5–6
  • On element-aware references, zero-shot GPT-3 is competitive with all fine-tuned PLMs and outperforms other models by a wide margin on BBC XSum.The comparison includes 175B GPT-3 and fine-tuned BART, T5, and PEGASUS.Source: Table 3 · p. 5, §3.2 · p. 5

3.1 Experimental Setup

  • Each 200-document test set compares zero-shot 175B GPT-3 and fine-tuned BART, T5, and PEGASUS against original and expert references using ROUGE, BERTSCORE, and human studies.The experiments use CNN/DailyMail and BBC XSum.Source: §3.1 · p. 5

3.3 Human Study

  • In the human study, GPT-3 summaries outperform almost all dataset-specific or model-generated summaries across dimensions, but remain below element-aware summaries.Annotators rated fluency, coherence, consistency, and relevance.Source: §3.3 · p. 6

4 Towards Element-oriented Summary: Chain-of-Thought Method

SumCoT prompts LLMs to extract core news elements before integrating them with source details to generate summaries. It improves automatic and human evaluations, while extraction quality depends on model scale.

4.1 Two-stage Pipeline

  • Stage 1 extracts Entity, Date, Event, and Result from the source using guiding questions; Stage 2 combines the source, questions, and answers to generate the summary.For XSum, the integration prompt asks for a one-sentence summary.Source: §4.1 · p. 6

4.2 Comprehensive Evaluation

  • SumCoT summaries contain more fine-grained elements than GPT-3 summaries without SumCoT in the Table 4 case comparisons.Table 4 highlights Entity, Date, Event, and Result spans in different colors.Source: §4.2 · p. 7, Table 4 · p. 7
  • +4.33 ROUGE-L on CNN/DailyMail and +4.77 ROUGE-L on BBC XSum over previous SOTA, with GPT-3 using SumCoT.The comparison is reported across metrics on element-aware test sets.Source: §4.2 · p. 7, Table 5 · p. 7
  • Human studies find that SumCoT improves all evaluated dimensions over standard zero-shot GPT-3, particularly coherence and relevance.Table 6 reports Fluency, Coherence, Consistency, and Relevance scores.Source: §4.2 · p. 7, Table 6 · p. 7

How does SumCoT affect summary writing?

5 Related Work and Discussion

Prior summarization datasets and reference-based metrics raise concerns about noisy summaries and evaluation alignment with human preferences. The paper applies chain-of-thought prompting to summarization, a setting it says prior work had not studied.

5.1 Summarization: Dataset and Evaluation

  • Standard summarization datasets have reported coherence, redundancy, and factuality noise, while reference-based metrics assess fit to dataset summaries and automatic scores may not align with human preference (Goyal et al., 2022).The paper notes that this concern also arises for evaluating LLM-generated summaries.Source: §5.1 · pp. 8–9

5.2 Chain-of-Thought Prompting for LLMs

6 Conclusion

The paper constructs expert-writing element-aware test sets for CNN/DailyMail and BBC XSum and reports promising zero-shot LLM summarization results. It also proposes a stepwise method focused on core news elements.

6 Conclusion

  • The expert-writing element-aware test sets target fuller assessment of general summarization capabilities across powerful language models.The sets are constructed for CNN/DailyMail and BBC XSum.Source: §6 · p. 9
  • The experiments demonstrate zero-shot summary-writing ability in LLMs, and the proposed CoT-based method focuses on core news elements to generate summaries step by step.The authors hope this work inspires research on LLMs mimicking human writing processes in open-ended generation.Source: §6 · p. 9

Limitations

The test sets are smaller than the original datasets and do not fully account for domain balance; the method's outputs may be sensitive to chain-of-thought form.

Limitations

  • Time, labor, and financial limits kept the test sets smaller than the originals, and domain balance is not fully considered, though uniform writing style may partly alleviate this.The authors identify larger, better-balanced test sets as a boundary of the current work.Source: Limitations · p. 9
  • Because generative tasks are more open-ended than pure reasoning, generated summaries may be more sensitive to chain-of-thought form, which the authors identify for further optimization.Source: Limitations · p. 9

Ethics Statement

The paper reports its ethical safeguards and the procedures used for its materials and human study. It also records the sources and licenses of its artifacts and packages.

Ethics Statement

  • The authors used public source documents and neutral, task-specific prompts; no toxic texts appeared in their experiments.They note prior work on potential toxicity in LLM-generated content and describe this as the first application of chain-of-thought to open-end generation tasks.Source: Ethics Statement · p. 9

A.1 Main Experiment

  • Table 9 lists the sources and licenses of the artifacts and packages used in the paper.The passage identifies the table’s scope but does not provide individual sources or license details.Source: §A.1 · p. 13

A.2 Human Study

  • Three privately contacted annotators conducted the human studies, with their participation voluntary and payment slightly above the crowd-sourced hourly rate.The annotators had backgrounds in computer science, film studies, and journalism and communication.Source: §A.2 · p. 13

B Abstraction and Faithfulness analysis for Summaries

The paper contrasts dataset-specific summaries’ abstraction and faithfulness with expert-written element-aware summaries, then illustrates how SumCoT changes GPT-3 summaries. The examples show SumCoT summaries covering more extracted fine-grained elements.

B Abstraction and Faithfulness analysis for Summaries

  • Dataset-specific summaries can have more novel n-grams yet sacrifice factual correctness, producing what the authors call fake high abstraction.The paper contrasts these summaries with element-aware summaries in abstraction and faithfulness.Source: B Abstraction and Faithfulness… · p. 13

C.1 Learn How SumCoT Works

  • GPT-3 summaries using SumCoT cover many fine-grained extracted elements absent from standard zero-shot summaries.The comparisons show GPT-3 outputs before and after SumCoT, with core elements highlighted.Source: §C.1 · p. 13

C.2 Error Analysis for Element Extractions

The error analysis identifies date hallucination and redundant extraction as distinct issues. The authors also compare GPT-3 model sizes using element-extraction F1 scores.

C.2 Error Analysis for Element Extractions

  • Date extraction errors arise when dates are absent or require reasoning from relative dates, and Date F1 is lower than for the other elements.The authors note that models may supply false dates when none are present and may fail on reasoning such as converting “two years ago” into a calendar year.Source: §C.2 · p. 13
  • LLMs often extract source-faithful but unimportant elements, which the authors identify as a source of lower Precision than Recall.They report this Precision–Recall pattern for almost every element.Source: §C.2 · p. 13

C.3 Ablation Study of GPT-3 Model Size

  • The GPT-3 size study compares 0.3B-, 1.3B-, 6.7B-, and 175B-parameter versions, with their extraction F1 curves shown in Figure 5.The supplied passages do not state which model size performs best.Source: §C.3 · p. 14

D Random Sample Presentation

This section presents examples comparing summaries, extraction errors, and GPT-3 model versions. The samples include source documents, reference and expert-written summaries, and GPT-3 outputs.

D Random Sample Presentation

B Abstraction and Faithfulness analysis for Summaries

  • Tables 10–11 compare element-aware and dataset-specific summaries in abstraction and faithfulness, noting more hallucinated facts in dataset-specific summaries despite their higher novel n-gram percentage.The captions identify highlighted hallucinations but provide no additional quantitative values.Source: Table 10 · p. 15, Table 11 · p. 15

C.1 Learn How SumCoT Works

  • Tables 12–14 compare GPT-3 zero-shot summaries with and without SumCoT, highlighting Entity, Date, Event, and Result spans in separate colors.The captions also indicate that prompts appear in italics.Source: Table 12 · p. 16, Table 13 · p. 17, Table 14 · p. 17

C.2 Error Analysis for Element Extractions

  • Tables 15–19 present GPT-3 element-extraction errors, while Table 20 presents extraction results for different GPT-3 versions.The supplied captions identify these tables’ subjects but do not report specific outcomes.Source: Table 15 · p. 18, Table 16 · p. 19, Table 17 · p. 19

Related papers

Loading 2305.13412v1…