PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

arXiv:2311.09184v2cs.CLcs.LG

TL;DR

Generic summarization is well studied, but evidence on LLM performance in more controlled settings aligned with users’ needs is limited. The paper benchmarks instruction-controllable generation and evaluation using a curated dataset and human annotations. It finds persistent summary errors, weak evaluator alignment on individual summaries, and substantial performance gaps across LLMs.

  • Problem

    Generic summarization can be underconstrained, leaving many acceptable summaries without clear comparison criteria, while LLM performance in settings aligned with users’ real needs remains an open question.

    Source: §1 · p. 1

  • Method

    The authors define summarization with a source article and natural-language requirement, curate an evaluation-only dataset, collect multidimensional human judgments, and benchmark LLM-based evaluators.

    Source: §1 · p. 2, §2.1 · p. 2, §2.2 · p. 3

  • Results

    LLMs still make summary errors, evaluator methods do not align strongly with human judgments of individual summaries, and performance gaps exist across LLMs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors argue that instruction-controllable summarization is more realistic and may provide better usability, and call for future work in this direction.

    Source: §6 · p. 9

    The analysis uses human annotations for 100 examples, and the authors note that larger evaluation sets would support more statistically significant conclusions.

    Source: §7 · p. 10

Abstract

from arXiv · show

While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. To this end, we curate an evaluation-only dataset for this task setting and conduct human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. We then benchmark LLM-based automatic evaluation for this task with 4 different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities. We make our collected benchmark InstruSum publicly available to facilitate future research in this direction.

1 Introduction

Generic summarization can yield many acceptable summaries without clear comparison criteria, leaving practical information needs unmet. The paper studies instruction-controllable summarization and finds persistent generation errors and weak evaluator alignment.

1 Introduction

  • Generic summarization benchmarks show strong LLM performance, but underconstrained prompts leave many good summaries without clear comparison criteria and may miss practical usability needs.Prior work also reports similar human ratings across model capacities and low, preference-sensitive annotator agreement for strong systems.Source: §1 · p. 1
  • Instruction-controllable summarization takes a source article and natural-language summary requirement as input, allowing flexible combinations of information queries and formatting requirements.The task extends query-focused and aspect- or attribute-based controllable summarization.Source: §1 · p. 2
  • Human evaluation found large performance gaps among LLMs, and even GPT-4 made factual and other summary errors; no LLM evaluator aligned strongly with humans judging individual summaries.Automatic evaluation can also be biased by confounding factors such as summary length.Source: §1 · p. 2
  • The authors curated an evaluation-only dataset and collected human judgments of representative LLM summaries across overall quality, missing information, irrelevant information, and factual consistency.They aim for requirements that reflect readers’ actual information needs.Source: §1 · p. 2
  • The authors benchmarked 40 automatic evaluation methods combining protocols and LLM backbones, finding performance gaps across both and strong system comparisons but weak alignment on individual summaries.GPT-4-powered methods are among those that perform strongly when comparing summarization systems.Source: §1 · p. 2
  • INSTRUSUM provides multidimensional human annotations of LLM-generated summaries and is publicly available; the study also benchmarks automatic evaluation methods and highlights their limitations.The dataset is intended to support research on summarization and summarization evaluation.Source: §1 · p. 2

2 Human Annotation Collection

The authors build INSTRUSUM through dataset construction and human evaluation, then use it to assess LLM summaries against instruction-controllable requirements. Their findings show substantial differences among systems, persistent factual errors, and practical constraints on annotation.

2.1 Dataset Creation

  • INSTRUSUM comprises 100 article-requirement pairs from XL-Sum, selected for abundant, complex information and around 1,000–1,200 words; requirements simulated readers’ differing informational needs and structural or formatting preferences.GPT-4 generated candidate requirements to increase diversity, but they were not frequently used.Source: §2.1 · p. 3

2.2 System Output Evaluation

  • Human evaluation covers 500 summary-level annotations across overall quality, missing information, irrelevant information, and factual consistency.Annotators ranked summaries from best to worst; for factual consistency they identified inconsistent spans, and for overall preference they explained their rankings.Source: §2.2 · p. 3
  • Crowd annotations achieved 88.4% accuracy on factual-error spans, while annotators showed moderately high agreement on most dimensions.The authors note that low agreement on factual consistency may reflect the sparsity of errors in the dataset.Source: §2.2 · p. 4
  • Annotation scale was difficult to expand: only around 5% of qualification-test participants met the task standard, and each annotation took about 30 minutes.The authors attribute sample-size constraints to both budget limits and the intensive labor required.Source: §2.2 · p. 4

2.3 Are LLMs Good at Ins-Controllable Summarization?

  • GPT-4 significantly outperforms GPT-3.5 models, while text-davinci-002 performs worse than models fine-tuned with reinforcement learning from human feedback.Factual error rates are 36% for text-davinci-002 and 14% for gpt-4-0314.Source: §2.3 · pp. 4–5
  • The hybrid LLM-human summary surpasses GPT-4 only on irrelevant information, the most challenging dimension for current LLMs and automatic evaluators.The authors suggest GPT-4 is close to human-level performance, especially when annotators only edit an LLM summary.Source: §2.3 · p. 5
  • Human annotators’ missing-information judgments have a higher influence on overall quality than their irrelevant-information judgments.This pattern suggests annotators favor comprehensive over concise summaries.Source: §2.3 · p. 5

3 Are LLMs Good at Ins-Controllable Summary Evaluation?

The section benchmarks 40 LLM-based evaluators across protocols and backbones against human judgments on instruction-controllable summaries. Evaluators differ substantially, and their system-level performance does not translate into reliable judgments of individual summaries.

3.1 LLM-based Evaluation Protocols

  • The benchmark tests four protocols—LLMScore, LLMEval, LLMCompare, and LLMRank—using 11 backbones across three dimensions, yielding 40 evaluation methods.Factual-consistency evaluation was excluded; some OpenAI models could not be tested with LLMScore because their APIs did not provide log-likelihoods.Source: §3.1 · p. 5, §3.2 · p. 5, §3.3 · p. 6

3.3 Result Analysis

  • For overall-quality pairwise comparison, gpt-4-0314 with LLMCompare achieved 0.277 agreement with human evaluation, below human inter-annotator agreement of 0.4428.Figure 2 summarizes average evaluator performance across quality dimensions and protocols.Source: §3.3 · p. 7, Figure 2 · p. 7
  • LLM-based evaluators have much higher system-level than summary-level correlations, and LLMCompare is almost always better than LLMRank.GPT-4 consistently outperforms other backbones; the best protocol can vary by backbone.Source: §3.3 · p. 7
  • Irrelevant Information is harder for LLM-based evaluators than Missing Information, indicating stronger performance on recall- than precision-oriented evaluation.The evaluation covers overall quality, missing information, and irrelevant information, but not factual consistency.Source: §3.2 · p. 5, §3.3 · p. 7
  • All evaluated LLMs show low summary-level consistency across LLMCompare and LLMEval, while gpt-4-0314 has high system-level consistency.These two protocols were selected for the consistency analysis as the most reliable.Source: §3.3 · p. 7, Table 7 · p. 7
  • Generation and evaluation performance do not always align: text-davinci-003 generates better summaries than gpt-3.5-turbo-0301 but evaluates them worse.Despite its promising generation, gpt-4-0314 does not outperform its generation performance when used to rerank candidate summaries.Source: §3.3 · pp. 7–8, Table 8 · p. 7

4 Can We Automate Ins-Controllable Summarization Benchmarking?

The authors use GPT-4 with pairwise comparison to benchmark 11 systems against gpt-4-0314, but find that length bias can undermine the resulting comparisons.

4.1 Evaluation Settings

  • GPT-4 with LLMCompare is used to compare 11 LLMs against gpt-4-0314 on the 100 human-evaluated examples, reducing benchmarking cost.The authors selected this evaluator as their best-performing method and used GPT-4 as the baseline.Source: §4.1 · p. 8

4.2 Result Analysis

  • GPT-4’s evaluation is positively correlated with the length oracle across quality dimensions, making current LLM-based benchmarking unreliable; with length controlled, no evaluated LLM clearly outperforms gpt-4-0314.The authors report that human evaluation is less prone to summary-length bias than GPT-4 evaluation.Source: Table 9 · p. 8, §4.2 · p. 8
  • GPT-4’s evaluation suggests Llama-2 models can outperform gpt-4-0314, but the authors suspect this overestimates their performance because longer summaries tend to be favored.Figure 3 compares model evaluation results and summary-length ratios relative to gpt-4-0314.Source: §4.2 · p. 8

5 Related Work

The paper situates its benchmark among work on summarization benchmarks, instruction following, and LLM-based evaluation. It emphasizes instruction-driven controllability and a multidimensional evaluation of summary quality.

Summarization Benchmarks

  • Prior summarization benchmarks aggregate model outputs and annotate quality dimensions, while this work presents a benchmark task intended to challenge current LLMs and support comparison.Related work also includes LLMs in benchmark construction and studies conditional text generation.Source: §5 · p. 9

Instruction-Following Evaluation

  • The task extends query-focused summarization by allowing information queries to combine with other user requests, such as output format.The paper's human evaluation is multidimensional and emphasizes summarization capability, unlike a related instruction-summary dataset focused only on instruction following.Source: §5 · p. 9

LLM-based Automatic Evaluation and Its Meta-Evaluation

  • Although LLMs show promise in automatic evaluation, prior work reports biases and failures to align with human judgments; this study meta-evaluates diverse protocols and backbones for instruction-controllable summarization.The cited failures include evaluation of close-performing systems and adversarial examples.Source: §5 · p. 9

6 Conclusions

The paper benchmarks LLMs for instruction-controllable summary generation and evaluation and introduces INSTRUSUM. It concludes that substantial errors, unreliable evaluation, and performance gaps remain.

6 Conclusions

  • LLMs still make considerable summary errors and cannot reliably evaluate candidate summaries, with large performance gaps across model generations in both generation and evaluation.The authors characterize instruction-controllable summarization as a more realistic setting that they believe can provide better usability.Source: §6 · p. 9

7 Limitations

The authors identify limits in the size of their human-annotated evaluation set and in evaluating factuality, and provide prompt templates, annotation details, and model-index information.

7 Limitations

  • The analysis covers 100 annotated examples; larger collections could support more statistically significant conclusions but require substantially more time and budget.The authors encourage applying their protocol to expand the evaluation set.Source: §7 · p. 10
  • Factuality evaluators were not meta-evaluated because factual errors were sparse and subtle, requiring a larger collection to reveal significant patterns.A larger evaluation of factual consistency and error types is left for future work.Source: §7 · p. 10

A.1 Prompts in Human Annotation Collection

  • The human-annotation prompts use GPT-4 to generate candidate summary requirements, with article-specific prompts requesting specific requirements and requirement-based summaries.The appendix supplies example requirements and the templates for generating requirements and summaries.Source: §A.1 · pp. 13–14

A.2 Prompts for LLM-based Automatic Evaluation

  • Automatic-evaluation prompts cover ranking, pairwise comparison, scoring, and yes/no evaluation; ranking and comparison prompts ask for explanations before decisions.The authors also describe chain-of-thought-style prompts for LLMRank and LLMCompare.Source: §A.2 · pp. 14–15

B Crowd-Annotation Details

  • Crowd annotation used US- or UK-based MTurk annotators, with an average hourly salary of about $20; Figures 4–6 show instructions, input, and result collection.Table 10 reports pairwise comparisons of initial and hybrid summaries and states that 37% were identical because no edits were made.Source: B Crowd-Annotation Details · p. 15, Figure 4 · p. 16, Figure 5 · p. 17

C OpenAI’s Model Index

  • The model-index appendix reports supervised fine-tuning on human demonstrations and highly rated model samples for text-davinci-002, and PPO with comparison-trained reward models for text-davinci-003.The authors say the information came from an OpenAI blog post.Source: C OpenAI’s Model Index · p. 19

D Fine-grained Analysis of Hybrid LLM-Human Summaries

The hybrid LLM-human summaries show mixed results across quality dimensions, with shorter summaries offered as a possible explanation for weaker performance on two dimensions.

D Fine-grained Analysis of Hybrid LLM-Human Summaries

  • Hybrid summaries outperform initial LLM summaries on irrelevant information and factual consistency, but underperform on overall quality and missing information.The comparison is reported in Table 10.Source: D Fine-grained Analysis of Hybrid… · p. 19
  • The authors attribute the pattern partly to more deletions than additions, since initial LLM summaries more often contain irrelevant than missing information; shorter hybrids may be less favored.The proposed explanation concerns overall quality and missing information.Source: D Fine-grained Analysis of Hybrid… · p. 19

E Factual Error Examples

The examples illustrate that factual errors flagged by annotators can be nuanced, pairing article excerpts with specific summary error spans.

E Factual Error Examples

  • Example 1 juxtaposes an article’s 2013 planning permission and funding plan, followed by a gold-price collapse, with a summary stating only that permission was obtained.Source: E Factual Error Examples · p. 19

Example 2

  • Example 2 pairs an account of a mine changing hands several times and being up for sale by 2006 with a summary making the same claims.Source: Example 2 · p. 19

Example 3

  • Example 3 gives WHO-information trust as about one-third of Republicans versus 80% of Democrats, and the summary states 33% versus 80%.Source: Example 3 · p. 19

Example 4

  • Example 4 contrasts experts’ expectation of very few or zero extra childhood cancers with a summary attributing Fukushima thyroid cancer cases to no nuclear-disaster link.Source: Example 4 · pp. 19–20

Example 5

  • Example 5 pairs Chum Mey’s account of confessing to end torture with a summary saying he was tortured until confessing to spying for the US and Russia.Source: Example 5 · p. 20

F Detailed Evaluation Settings of LLM-based Evaluation Methods

The evaluation settings compare human and LLM judgments through system-level and summary-level rank correlations, converting protocol outputs to rankings and shuffling order for selected protocols.

F Detailed Evaluation Settings of LLM-based Evaluation Methods

  • System-level correlation compares the lists of average system scores assigned by two evaluation methods using a correlation measure such as Kendall rank correlation.Source: F Detailed Evaluation Settings of… · p. 20
  • Summary-level correlation averages correlations between the two methods’ scores for summaries generated by different systems on each example; N is the evaluation-dataset size.Source: F Detailed Evaluation Settings of… · p. 20
  • The authors use Kendall rank correlation, convert direct scores to rankings, aggregate pairwise-comparison wins, losses, and ties into system rankings, and shuffle order for LLMRank and LLMCompare.Source: F Detailed Evaluation Settings of… · p. 20

G Length Bias in LLM-based Evaluation

Length controls remove Llama-2’s apparent edge over GPT-4, while requirement-specific summaries strongly beat generic ones. The authors also define a long-context evaluation setting and extend their experiments into a leaderboard of automatic evaluators.

G Length Bias in LLM-based Evaluation

  • With summary lengths controlled, no compared LLM outperforms GPT-4 on overall quality, and Llama-2 loses its clear advantage; system-pair results are not directly comparable.The comparison uses summary pairs within the 20th percentile of length differences.Source: G Length Bias in LLM-based Evaluation · p. 20, Table 11 · p. 21

H Comparing Generic and Requirement-Specific Summaries

  • Requirement-specific summaries win over generic summaries 97% of the time for gpt-4-0314 and 96% for gpt-3.5-turbo-0301 on overall quality.Generic summaries were not preferred by gpt-4-0314 despite their longer average length, and their similarity to requirement-specific summaries was relatively low.Source: H Comparing Generic and… · pp. 20–21

I INSTRUSUMEVAL Leaderboard

  • The authors expand their experiments into a large-scale leaderboard of automatic evaluators using the human annotations collected in this work.Source: I INSTRUSUMEVAL Leaderboard · p. 21

I.1 Evaluation Setting

  • INSTRUSUMEVAL evaluates pairwise judgments with source articles and requirements combined into instructions averaging over 1,000 words, making it a long-context instruction-following benchmark.Evaluator accuracy against human annotations and bidirectional self-consistency are assessed using accuracy and Krippendorff’s alpha agreement.Source: §I.1 · pp. 21–22

I.2 Data Filtering

  • Filtering 1,000 pairwise annotations from 100 rankings across five systems to cases with perfect human agreement and no ties leaves 411 instances.Source: §I.2 · p. 22

I.3 Results

The INSTRUSUMEVAL leaderboard reports substantial performance differences among LLMs of different capacities and compares their judgments with human annotations and their own bidirectional predictions.

I.3 Results

  • Table 14 benchmarks 41 LLMs and shows large performance gaps across capacities, reporting human-alignment and bidirectional self-consistency measures.Accuracy and agreement are measured against human annotations; self-accuracy and self-agreement compare the LLMs’ bidirectional predictions.Source: §I.3 · p. 22, Table 14 · p. 23
Loading 2311.09184v2…