PaperlayerView exact revision on arXiv

Source-linked AI summary

Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles

Kung-Hsiang Huang, Philippe Laban, Alexander R. Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu

arXiv:2309.09369v2cs.CL

TL;DR

Multi-document news summaries have often emphasized consensus, leaving diverse information across sources underexplored. The paper defines a diversity-summarization task, builds a human-validated dataset, and analyzes LLM summaries and evaluation. GPT-4 still covers only about 37% of diverse information with optimized prompts, and the authors identify prompt sensitivity as an unexamined factor in their evaluations.

  • Problem

    Prior multi-document news summarization has typically emphasized information sources agree on, while diverse information across articles remains underexplored.

    Source: §1 · p. 1

  • Method

    The authors define a multi-document diversity-summarization task, construct human-validated QA references for DIVERSESUMM, and analyze LLM summaries and GPT-4-based evaluation protocols.

    Source: §1 · pp. 1–2

  • Results

    GPT-4 covers about 37% of diverse information with optimally designed prompts, and the study analyzes bias and correlation in GPT-4 evaluations against human assessments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors provide recommendations for using GPT-4-based metrics to assess coverage and faithfulness in this task domain.

    Source: §1 · p. 2, §6 · p. 9

    The study did not investigate how different prompts influence LLM performance, so prompt sensitivity could lead to slightly different outcomes.

    Source: §8 · p. 10

Abstract

from arXiv · show

Previous research in multi-document news summarization has typically concentrated on collating information that all sources agree upon. However, the summarization of diverse information dispersed across multiple articles about an event remains underexplored. In this paper, we propose a new task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, we outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. The dataset includes 245 news stories, with each story comprising 10 news articles and paired with a human-validated reference. Next, to enable consistent automatic evaluation, we conducted a comprehensive analysis to pinpoint the position and verbosity biases when utilizing Large Language Model (LLM)-based metrics for evaluating the coverage and faithfulness of summaries. Through correlation analyses, we outline the best practices for effectively using automatic LLM-based metrics on the DiverseSumm dataset. Finally, we study how LLMs summarize multiple news articles by analyzing which type of diverse information LLMs are capable of identifying. Our analyses suggest that despite the extraordinary capabilities of LLMs in single-document summarization, the proposed task remains a complex challenge for them mainly due to their limited coverage, with GPT-4 only able to cover under 40% of the diverse information on average.

1 Introduction

The paper introduces multi-document diversity summarization to capture differing viewpoints that consensus-focused summaries leave out. It builds a human-validated QA-reference dataset and studies LLM coverage and evaluation, finding that GPT-4 covers only about 37% of diverse information even with optimized prompts.

1 Introduction

  • The proposed Multi-document Diversity Summarization task targets diverse information across news articles, unlike prior multi-document summaries that often reflect only consensus.The paper frames the task as faithfully illuminating differing information in multiple sources.Source: §1 · p. 1
  • Diverse information is represented as questions answerable by multiple sources whose extracted answers express varied opinions or perspectives.The authors choose QA references for their granularity and reliability advantages in evaluation.Source: §1 · p. 1
  • DIVERSESUMM contains 245 news-story clusters of 10 articles each, built with GPT-3.5-Turbo question generation, answer extraction and clustering, filtering, and human validation.Each story averages 2.49 questions, with 3.41 answer clusters per question.Source: §1 · p. 1
  • GPT-4 covers about 37% of diverse information with optimally designed prompts, while the study also analyzes GPT-4 evaluation biases and correlations with human assessments.The authors provide recommendations for applying GPT-4-based metrics to coverage and faithfulness.Source: §1 · p. 2

2 Task

The task asks for a natural-language summary of diverse information in a cluster of 10 articles about one event. Its assessment centers on whether summaries are faithful to the articles and cover the reference answers.

2 Task

  • Each MDDS input is a cluster of 10 articles about one event, and the output is a natural-language summary capturing diverse information across them.References are question-answer pairs that must be answerable by enough sources and exhibit diverse opinions or perspectives.Source: §2 · p. 2
  • Evaluation focuses on faithfulness to source facts and coverage of reference answers, rather than qualities such as compression ratio or coherence.The paper identifies faithfulness and coverage as major challenges for pretrained-transformer summarization models.Source: §2 · pp. 2–3

3 Data Collection

DIVERSESUMM is built through automated discovery and curation of diverse question-answer information, followed by human validation. The curation changes target question quality and answer recall, while the final dataset contains 245 stories with 10 articles each.

3.1 Automatic Data Curation

  • The curation pipeline generates questions, extracts article answers, clusters them semantically, and filters invalid or non-diverse question-answer pairs.Its four modifications include two-stage question generation, article-context QA, post-processing, and GPT-3.5-Turbo.Source: §3.1 · p. 3
  • The dataset is collected from Google News stories, with articles selected from sources covering each story and most sampled articles published in March 2023.The authors initially picked 400 recent news stories; most articles were beyond GPT-3.5-Turbo's September 2021 knowledge cutoff.Source: §3.1 · p. 3
  • Valid questions require answers from at least 30% of sources and assorted responses, while single-article question generation yielded only 10 qualifying questions from 200.The authors used a medium-length article per story to generate 20 questions before extracting answers across sources.Source: §3.1 · pp. 3–4
  • 85 questions linked to sufficient answers (+102.4%) and 19 valid questions (+90.0%) resulted from the revised strategy in a manual assessment of 10 stories.The paper attributes the change to its novel question-generation approach compared with the prior method.Source: §3.1 · p. 4
  • Article-level GPT-3.5-Turbo answer extraction reaches 64.6% recall, versus 43.8% for RoBERTa, so the authors select it despite RoBERTa's higher precision.They reason that excess answers are easier to filter than missed answers are to recover.Source: §3.1 · p. 4

3.2 Human Validation

  • 245 news stories with 10 articles each remain after post-processed question-answer pairs undergo human validation.Annotators check whether an answer addresses its question and appears in at least one article.Source: §3.2 · p. 4

4 Analysis

Human evaluation finds that LLMs are generally faithful but cover too little diverse information. The analysis also identifies biases in GPT-4 evaluation and differences in which article positions and question types models summarize.

4.1 RQ 1: How proficient are LLMs in summarizing diverse information from multiple news articles?

  • All four evaluated LLMs generally achieve high faithfulness but insufficient coverage, making MDDS challenging even for state-of-the-art models.The authors describe DIVERSESUMM as a challenging test bed for LLMs.Source: §4.1 · p. 5

4.2 RQ 2: What are the pitfalls and best practices when leveraging GPT-4 as the evaluation metric for our task?

  • GPT-4 favors the second summary in pairwise judgments and shorter summaries in single-answer grading, while pairwise comparison significantly lessens verbosity bias.The second-position bias is particularly pronounced for summaries of similar quality.Source: Table 3 · p. 5, Table 4 · p. 5, §4.2 · p. 6
  • For faithfulness, both-way pairwise comparison is recommended when budget permits; otherwise, GPT-4 single-answer Likert grading is the recommended alternative.For coverage, single-answer Likert grading has the highest correlation with human judgments; pairwise comparison is computationally demanding.Source: §4.2 · pp. 6–7, Table 5 · p. 6

4.3 RQ 3: Do LLM exhibit coverage bias when performing MDDS?

  • Higher-faithfulness LLMs draw more from the first and last articles than the middle ones, whereas lower-faithfulness models show no clear positional pattern.Figure 3 depicts this U-shaped pattern for more faithful models.Source: §4.3 · p. 7, Figure 3 · p. 7
  • Coverage is better for “Why” and “Where” questions than for “How” and “What” questions, which require connecting events or interpreting implications.Figure 4 encodes higher coverage in blue and lower coverage in red.Source: §4.3 · p. 7, Figure 4 · p. 7

5 Related Work

Prior multi-document summarization work includes extractive, abstractive, and compression approaches, and LLM-based methods. Existing datasets vary in scale and domain, while the paper distinguishes its focus on diverse information from consensus summarization.

5.1 Multi-document Summarization

  • Multi-document summarization approaches include extractive methods, abstractive methods, and multi-sentence compression.The cited work includes extractive approaches such as Radev et al. (2000) and abstractive approaches such as Fabbri et al. (2019).Source: §5.1 · p. 9
  • Recent LLM approaches summarize multiple documents by selecting important sentences before summarizing or iteratively improving summaries with checklist guidance.The passage cites Bhaskar et al. (2023) and Zeng et al. (2023), respectively.Source: §5.1 · p. 9

5.2 MDS Datasets

  • Existing MDS datasets span news, Wikipedia, and scientific domains, with cluster scale varying from about 50 or 100 to 56K articles.MULTINEWS has 56K news clusters averaging fewer than three articles per cluster; DUC and TAC have approximately 50 and 100 clusters, respectively.Source: §5.2 · p. 9
  • Whereas prior MDS datasets task models with summarizing consensus information, this work focuses on diverse information across articles.The cited datasets include DUC, TAC, MULTINEWS, AUTO-HMDS, WCEP, MULTI-XSCIENCE, and MS^2.Source: §5.2 · p. 9

6 Conclusion

The paper introduces a task and QA-based dataset for summarizing diverse information across news articles. Human evaluation finds that LLMs are highly faithful but still struggle with coverage, alongside analyses of evaluation bias and model behavior.

6 Conclusion

  • The paper introduces Multi-document Diverse Summarization and DIVERSESUMM, using a QA-based pipeline to support the task.The dataset name is split across the supplied passage text; the conclusion identifies it as DIVERSESUMM.Source: §6 · p. 9
  • Human evaluation finds that LLMs are highly faithful on the task, but achieving high coverage remains challenging even for GPT-4.The authors describe this as both a challenge and an opportunity for MDDS.Source: §6 · p. 9
  • The paper analyzes evaluation bias and correlation with human assessments, recommends evaluation protocols, and studies the types of information LLMs summarize.The authors present these findings as resources for future work on diverse news information.Source: §6 · p. 9

7 Ethical Considerations

The paper describes participant protections for annotation and human evaluation, including fair compensation, autonomy during participation, and anonymity safeguards.

7 Ethical Considerations

  • Participants were fairly compensated, with compensation details provided in Appendix A.Source: §7 · p. 9
  • Participants could set their own pace, raise concerns through open communication, and withdraw at any time without repercussions.Source: §7 · p. 9
  • Annotations were kept anonymous by avoiding personally identifiable information.Source: §7 · p. 10

8 Limitation

The authors identify dataset size, prompt sensitivity, and domain transfer as limitations of the study.

8 Limitation

  • The dataset covers only a relatively small fraction of online news, reflecting the time and cost of annotating multiple articles.The authors attribute those costs to the need for annotators to read and understand multiple articles, unlike single-document annotation.Source: §8 · p. 10
  • The study does not test prompt sensitivity, so different prompts could produce slightly different evaluation outcomes.The authors manually optimized prompts but did not measure their exact influence on LLM performance.Source: §8 · p. 10
  • Because the dataset contains online news, the findings may not generalize to other multi-document domains such as academic or legal documents.The authors say generalizability beyond news requires further analysis.Source: §8 · p. 10

A Are our findings in §4.2 still reproducible after a GPT-4 update every two months?

The appendix argues that evaluation principles may carry across GPT updates, while noting that new tests would be useful. It also describes worker qualification and human checks for QA validity.

A Are our findings in §4.2 still reproducible after a GPT-4 update every two months?

  • New tests would be ideal after model updates, although the authors argue their evaluation principles can likely be adapted to newer GPT versions.They describe the potential need for updated validation as common in evolving fields.Source: A Are our findings in §4.2 still… · p. 13

B.1 Worker Qualification

  • Workers had to meet experience and approval thresholds, pass three qualification rounds, and show sufficient agreement with manually annotated samples.The study selected 16 workers who passed all three rounds to conduct its evaluations and annotations.Source: §B.1 · p. 13

B.2 Annotating QAs

B Human Annotation

  • Human annotators check post-processed QA pairs for whether answers address their questions and are supported by at least one article.Answers need not match an article word for word; the information must be present and address the paired question.Source: B Human Annotation · p. 13, §B.2 · p. 14

B.3 Coverage Evaluation

The appendix specifies how annotators judge coverage and faithfulness, then reports agreement levels for these judgments and reference annotations.

B.3 Coverage Evaluation

  • A statement counts as covered when its information is mentioned or supported by the corresponding summary, even without exact wording.Annotators compare each statement with its corresponding summary.Source: §B.3 · p. 14

B.4 Faithfulness Evaluation

  • A summary sentence is faithful when at least one source article supports it; nonsense sentences are rated unfaithful.Annotators compare each generated sentence with the source articles and mark unsupported sentences “Not Present.”Source: §B.4 · p. 14

B.5 Inter-annotator Agreement

  • Krippendorff’s Alpha was 0.61 for faithfulness, 0.60 for coverage, and 0.69 for reference annotations.The authors characterize these agreement levels as moderate to high.Source: §B.5 · p. 15

C LLM Prompts

The appendix presents prompts used for reference annotation, summarization, and evaluation. Summarization prompts vary by model context length and were manually engineered to encourage more comprehensive outputs.

C LLM Prompts

  • The appendix displays experimental prompts, with boldface marking placeholders.The section introduces the prompt materials used in the experiments.Source: C LLM Prompts · p. 15

C.1 LLM Prompts for Reference Annotation

  • Reference annotation uses prompted ChatGPT for question generation, question answering, and post-processing.The appendix points to separate figures for each component’s prompt.Source: §C.1 · p. 15

C.2 LLM Prompts for Summarization

  • Summarization uses different prompts for long-context and standard LLMs because standard models cannot process all input articles in one context.The appendix shows one template for long-context models and two for standard models.Source: §C.2 · p. 15
  • The authors report that adding “Don’t worry about the summary being too lengthy” to GPT-4’s prompt helps generate more comprehensive summaries.They say their prompts were meticulously engineered and generally produced summaries with higher coverage.Source: §C.2 · p. 15
  • The authors use summary length as a proxy for coverage and report that their final prompt significantly increased output length over the initial prompt.Figure 12 presents the length comparison between the final and initial prompts.Source: §C.2 · p. 15

C.3 LLM Prompts for Evaluation

  • The appendix also displays the GPT-4 prompts used for evaluation.These are presented in the evaluation-prompt subsection.Source: §C.3 · p. 15

D LLM Bias Analysis

The bias analysis finds GPT-4-based evaluation sensitive to summary order and length: pairwise comparison mitigates verbosity bias, while single-answer grading favors shorter summaries. The section also presents topic and question distributions for DiverseSumm.

D.1 Position Bias

  • GPT-4-based pairwise evaluation significantly favors the second summary, especially when the summaries are similar in quality.This position bias appears in both coverage and faithfulness evaluation.Source: §D.1 · p. 16, Figure 23 · p. 22, Figure 24 · p. 23

D.2 Verbosity Bias

  • Pairwise comparison significantly mitigates verbosity bias, while single-answer grading prefers shorter summaries across models for both faithfulness and coverage.The preference for shorter summaries runs counter to the authors’ expectation that coverage evaluation would favor longer summaries.Source: §D.2 · p. 16, Figure 25 · p. 23

E Topic and Question Distribution

Related papers

This paper cites

Loading 2309.09369v2…