PaperlayerView exact revision on arXiv

Source-linked AI summary

BooookScore: A systematic exploration of book-length summarization in the era of LLMs

Yapei Chang, Kyle Lo, Tanya Goyal, Mohit Iyyer

arXiv:2310.00785v4cs.CLcs.AIcs.LG

TL;DR

Book-length summarization exceeds LLM context windows and lacks reliable evaluation because existing benchmarks may be contaminated and standard methods miss long-document coherence errors. The paper compares chunk-based prompting workflows and develops human-validated coherence evaluation with BooookScore. Its evaluations show systematic differences across models and workflows, including a coherence-detail trade-off between hierarchical merging and incremental updating.

  • Problem

    Book-length summarization lacks principled evaluation because existing benchmarks may contaminate modern LLMs, long-document coherence errors are understudied, and reliable automatic metrics are unavailable.

  • Method

    The paper evaluates two chunk-based prompting workflows using human annotations of summaries from newly published books and develops the reference-free BOOOOKSCORE metric.

  • Results

    Hierarchical merging generally produces more coherent but less detailed summaries than incremental updating, while GPT-4 and Claude 2 produce the most coherent summaries.

  • Takeaways & Limitations

    Reference-free BOOOOKSCORE enables systematic comparison of book-length summarization configurations without gold summaries.

  • Takeaways & Limitations

    The error taxonomy and BOOOOKSCORE use GPT-4 as their base model, which may introduce systematic biases.

Abstract

from arXiv · show

Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving $15K USD and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models. While LLaMA 2 falls behind other models, Mixtral achieves performance on par with GPT-3.5-Turbo. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by annotators.

1 INTRODUCTION

Book-length summarization concerns texts over 100K tokens, where context limits require chunking and combining partial summaries. The paper addresses evaluation gaps by introducing a human-validated coherence protocol, BooookScore, and systematic comparisons of models and prompting strategies.

  • Texts longer than 100K tokens require chunking and combining partial summaries because they exceed current LLM context windows.
  • Existing evaluation is limited by benchmark contamination, unexplored book-length coherence errors, and the absence of a reliable automatic metric.
  • The protocol uses 1193 human annotations from GPT-4 summaries of 100 recently published books and identifies eight frequent coherence-error types.
  • BOOOOKSCORE automatically detects the eight coherence errors and achieves human-level reliability without relying on gold summaries.
  • Hierarchical merging generally produces more coherent but less detailed summaries than incremental updating.
  • GPT-4 and Claude 2 produce the most coherent summaries, whereas LLaMA 2 performs substantially worse and fails to follow instructions.

2 BACKGROUND: SUMMARIZING BOOK-LENGTH TEXTS WITH LLMS

Because book-length documents exceed LLM context windows, the paper divides them into chunks and compares hierarchical merging with incremental updating. Both workflows repeatedly transform chunk-level summaries into a summary of the full book.

  • Hierarchical merging summarizes chunks separately and recursively merges their summaries until one full-book summary remains.
  • Both strategies repeatedly merge, update, or compress chunk-level partial summaries because the input length exceeds the model context window.
  • Book-length summarization first divides a document into smaller chunks that fit within an LLM context window.
  • Incremental updating maintains a global summary that is updated and compressed as the system processes chunks sequentially.

C ⌉where C < W is the length of each chunk.2

The paper contrasts hierarchical merging with incremental updating as two ways to summarize chunks when the full document exceeds the model context window. Their central difference is whether chunks are summarized independently or incorporated into a running summary.

  • Hierarchical merging: Hierarchical merging uses separate prompts to summarize chunks and recursively merge their summaries into a single book summary.
  • Incremental updating: Incremental updating processes chunks sequentially by revising and compressing a global summary as new text arrives.
  • Incremental updating: Hierarchical merging may introduce coherence errors because later chunks can be summarized without knowledge of earlier content.

3 EVALUATING COHERENCE OF BOOK SUMMARIES

The evaluation framework uses newly published books and reference-free, fine-grained human annotation to identify coherence errors in LLM-generated summaries. Annotators mark confusing spans, pose questions, and organize the resulting cases into an error taxonomy.

  • Corpus collection: The framework focuses on newly published books to reduce contamination from existing pretraining data and publicly available summaries.
  • Evaluation framework: Reference-free evaluation combines fine-grained units, information-seeking questions, and practices from document-understanding and summarization research.
  • Annotation protocol: Annotators read each generated summary, highlight confusing spans, and write questions describing the source of confusion.
  • Annotation protocol: The study uses four annotators, each reviewing 25 disjoint summaries, with roughly 30 minutes required per summary.
  • Error taxonomy: Error prevalence is normalized by the total number of sentences across all generated summaries.
  • Error taxonomy: The taxonomy contains eight coherence-error types, including causal omissions and salience issues that were not present in the referenced SNaC schema.

4 BOOOOKSCORE: AN AUTOMATIC EVALUATION METRIC

BOOOOKSCORE is a source- and reference-free automatic metric that uses an LLM to detect eight coherence-error types and measures the proportion of summary sentences without errors. Human validation supports its reliability, enabling scalable evaluation of book-length summarization configurations.

  • Metric design: BOOOOKSCORE prompts an LLM to identify instances of eight established coherence-error types in generated summaries.Definitions and examples from the error taxonomy are critical for achieving high annotation precision.
  • Validation: 78.2% vs. 79.7%: BOOOOKSCORE’s annotation precision matches that of human annotators in validation.The metric was then used to evaluate additional book-length summarization configurations.
  • Scalability: BOOOOKSCORE saved $15K USD and 500 hours of annotator time when evaluating many configurations.The broader evaluation reports coherence across prompting strategies, base LLMs, and chunk sizes.
  • Metric design: The metric is source-free and reference-free, so it does not require the input book or a gold summary.This design supports scalable evaluation when reference summaries are unavailable.
  • Metric definition: BOOOOKSCORE measures the proportion of sentences containing no coherence errors, treating each sentence as one unit of confusion.System-level scores are computed as the mean BOOOOKSCORE across summaries generated by that system.

5 SYSTEMATIC EVALUATION OF LLMS

The evaluation compares prompting strategies, base LLMs, and chunk sizes using BOOOOKSCORE, revealing trade-offs between coherence, detail, and model capability.

  • Claude 2 produces the most coherent summaries by BOOOOKSCORE, followed by GPT-4, while GPT-4 summaries are longer and more detailed across both strategies.The comparison includes GPT-3.5-Turbo, Mixtral-8x7B, and LLaMA2-7B-Inst.
  • Hierarchical summaries generally achieve higher BOOOOKSCORE than incremental summaries, likely because incremental updating requires more complex instructions.
  • 90.9 vs 90.3 BOOOOKSCORE: Claude 2 with an 88K chunk produces slightly more coherent incremental than hierarchical summaries.
  • 78.6 BOOOOKSCORE: Claude 2 incremental summarization with 2048-token chunks performs substantially worse than its 88K configuration.The result supports using fewer updating and compression steps for incremental summarization.
  • Mixtral-8x7B closely matches GPT-3.5-Turbo on both approaches, whereas LLaMA-2-7B-Instruct has the worst hierarchical score and fails at incremental updating.LLaMA 2 also produces significant repetition; Mixtral remains behind most closed-source models.
  • 83% vs. 11%: annotators prefer incremental summaries for detail, but prefer hierarchical summaries for structure, logical consistency, and overall quality.Overall preference is 54% for hierarchical summaries versus 44% for incremental summaries.
  • Qualitative analysis finds repetition and vagueness in GPT-4 and GPT-3.5-Turbo summaries, omissions in Claude 2’s 88K summaries, and faithfulness errors across configurations.

6 LIMITATIONS

The paper’s limitations concern the scope and cost of its evaluation: the taxonomy and metric are GPT-4-centered, while BOOOOKSCORE remains slower and more expensive than simpler automatic evaluation.

  • The error taxonomy is derived only from GPT-4 errors, and BOOOOKSCORE also uses GPT-4 as its base evaluator, potentially introducing systematic bias.The authors suggest a pool of LLM annotators could alleviate such biases.
  • BOOOOKSCORE is expensive and slow because it uses GPT-4 to evaluate summaries sentence by sentence with a long annotation prompt.It is nevertheless cheaper and faster than human evaluation.
  • BOOOOKSCORE does not weight different error types by relative importance because severity is ill-defined for many taxonomy categories.

7 RELATED WORK

Prior long-form summarization research has largely focused on documents shorter than 10K tokens, while this work addresses book-length narrative summarization.

  • Most prior long-form summarization work focuses on documents shorter than 10K tokens.BookSum is identified as the first published dataset including book-level source text.
  • BookSum’s book-level source text encouraged subsequent modeling efforts in book-length summarization.
  • The work relates to fine-grained evaluation protocols that annotate spans, error types, and severities, as well as protocols for salience and factual consistency.

8 CONCLUSION

The paper presents a systematic study of book-length LLM summarization, introducing a human coherence protocol and BOOOOKSCORE to compare prompting strategies and model choices. Its findings include a coherence–detail trade-off between hierarchical merging and incremental updating, while releasing the metric and annotations for future work.

  • Contributions: The study introduces a human evaluation protocol for coherence in book-length LLM summarization and releases annotated summaries with BOOOOKSCORE.The paper also identifies future directions including faithfulness evaluation, newer long-context models, and multilingual texts.
  • Prompting strategies: Book-length summarization splits documents into chunks, then repeatedly summarizes, merges, updates, or compresses partial summaries.Hierarchical merging recursively combines summaries, whereas incremental updating maintains a running global summary across chunks.
  • Prompting strategies: Hierarchical merging incorporates preceding summaries at the same level before recursively producing one summary for the book.This design is intended to mitigate context loss from independently summarizing chunks.
  • Prompting strategies: Incremental updating processes chunks sequentially and may require compression because the running summary tends to accumulate information rather than remove it.The method therefore uses a separate compression prompt when the global summary exceeds the maximum length.
  • Evaluation resources: The paper’s conclusion is supported by dataset and evaluation resources covering all books and 100 human-evaluated summary pairs.The cited tables document the book dataset and coarse-grained human evaluation.

D MORE EXPERIMENT DETAILS

The supplementary experiments document model configurations, preprocessing choices, and GPT-4 evaluation on SQuALITY. On that shorter-story benchmark, incremental updating slightly lowers ROUGE-L relative to the baseline, while qualitative analysis remains planned for BOOOOKSCORE.

  • Model configurations: GPT-4 summaries use Azure checkpoints, while Claude 2 summaries come from its API and Mixtral uses a Together-hosted checkpoint.LLaMA-2-7B-Instruct is fine-tuned on long-context summarization, and the reported Claude 2 summaries were obtained in September 2023.
  • Model configurations: LLaMA 2 uses a 4096-token context window, requiring truncation at final punctuation because it does not follow the word limit reliably.The experiments also omit post-processing for LLaMA 2 summaries because it would significantly alter their structure.
  • Preprocessing: The preprocessing extracts EPUB text without further automatic processing, so non-narrative content can enter summaries before GPT-4 removes it.The cleanup prompt targets material from tables of contents, acknowledgments, biographies, and other non-narrative sections.
  • SQuALITY experiments: Incremental updating lowers ROUGE-L by a small amount compared with the one-pass baseline on GPT-4 summaries of SQuALITY stories.The baseline has slightly greater overlap with the provided reference summaries.

E.1 QUALITATIVE ANALYSIS

Qualitative analysis contrasts incremental and baseline summaries and examines BOOOOKSCORE across length, error distributions, and annotation agreement. The reported evidence shows no discernible length correlation and indicates that GPT-4’s error labels can differ from human annotations.

  • Qualitative comparison: GPT-4 summaries can be better than human summaries, while incremental updating can introduce coherence errors.
  • BOOOOKSCORE analysis: There is no discernible correlation between BOOOOKSCORE and summary length for 100 GPT-4 incremental summaries using 2048-token chunks.
  • BOOOOKSCORE analysis: 0.015 is the standard deviation of bootstrapped BOOOOKSCORE means across 1000 resamples, suggesting consistency across random samples.The resamples draw 100 summaries with replacement from the GPT-4 incremental, 2048-token-chunk setting.
  • Error analysis: GPT-4’s error distributions vary significantly from human annotations despite high agreement in precision, with a strong propensity to label omission errors.

I EFFECTIVENESS OF EXISTING REFERENCE-FREE EVALUATION METRICS

The paper evaluates reference-free metrics and describes the human annotation framework and summarization prompts used in the broader experiments. BLANC and SUPERT show negligible model differences and contradict qualitative judgments in one comparison.

  • Metric effectiveness: BLANC and SUPERT differ by nearly negligible margins across models, making them less meaningful for book-level summary evaluation.
  • Metric effectiveness: BLANC and SUPERT rate Claude 2 88000 summaries below GPT-3.5-Turbo 2048 summaries, contradicting the qualitative analysis.
  • Human evaluation: The human evaluation asks whether annotators agree, partially agree, or disagree with span–question annotations concerning summary coherence.Questions must be relevant, unanswered in the summary, and connected to issues that would make the summary incoherent or obscure its main storyline.
  • Human evaluation: The evaluation mixes human and GPT-4 annotations while concealing their origins to reduce potential annotator bias.Annotators evaluated separate sets of books for the annotation and evaluation tasks.
  • Human evaluation: Agreement includes both agreement and partial agreement when computing precision, while disagreement means the span is not confusing.
  • Prompting procedures: The study uses recursive merging and sequential updating prompts that ask summaries to preserve key events, characters, settings, objectives, and motivations.The prompts also request coherent chronological narratives within target word limits.
Loading 2310.00785v4…