Source-linked AI summary
BookSum: A Collection of Datasets for Long-form Narrative Summarization
Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, Dragomir Radev
TL;DR
Existing summarization datasets are often short and fact-centric, limiting long-range narrative challenges. BOOKSUM addresses this gap with literary documents and highly abstractive summaries at paragraph, chapter, and book levels, then evaluates extractive and abstractive baselines. The dataset establishes a long-narrative benchmark whose use may require substantial computational resources.
Problem
Existing summarization datasets largely contain short documents with limited long-range causal and temporal dependencies and strong layout or stylistic biases.
Method
BOOKSUM collects literary documents with human-written abstractive summaries aligned at paragraph, chapter, and book levels, and benchmarks extractive and abstractive architectures.
Results
BOOKSUM provides a dataset designed to test long-document processing, distributed causal and temporal understanding, rich discourse handling, and abstractive compression.
Takeaways & Limitations
The hierarchical BOOKSUM collection supports incremental research on long-narrative summarization and comparison with future methods.
Takeaways & Limitations
Training and evaluating neural models on BOOKSUM may require substantial computational resources because its source documents are long.
Abstract
from arXiv · showhide
The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future generations of text summarization systems. We address these issues by introducing BookSum, a collection of datasets for long-form narrative summarization. Our dataset covers source documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of our dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. To facilitate future work, we trained and evaluated multiple extractive and abstractive summarization models as baselines for our dataset.
1 Introduction
Existing summarization datasets largely use short, fact-centric documents with limited long-range dependencies and layout variation. BOOKSUM introduces long-form literary data with abstractive summaries across three granularities and evaluates baseline systems on the resulting challenges.
- Motivation: Most summarization datasets focus on short news, scientific, or legal documents with explicit facts and constrained layout or style.These characteristics limit long-range causal, temporal, interpretive, and discourse challenges for summarization systems.
- Contribution: BOOKSUM covers stories, plays, and novels with highly abstractive, human-written summaries.The collection targets literature-domain documents rather than the predominantly newswire, scientific, and legal sources used in existing datasets.
- Contribution: The dataset challenges models to process texts spanning hundreds of pages and capture distributed causal, temporal, and discourse dependencies.Its discourse structures include parallel plots and shifts between narration and dialogue, while summaries are highly abstractive and compressive.
- Contribution: BOOKSUM provides paragraph-, chapter-, and book-level examples with increasing input and summary lengths.Paragraph examples use hundreds of words and short summaries; chapter examples span several pages; book examples span up to hundreds of pages.
- Evaluation: The authors evaluate extractive and abstractive state-of-the-art architectures on the newly introduced long-narrative summarization task.The evaluation is intended to demonstrate BOOKSUM’s challenges and establish baselines for future research.
2 Related Work
Earlier work addressed fiction and book summarization, but coverage and abstraction remained limited. BOOKSUM extends this line with larger aligned resources at paragraph, chapter, and full-book granularity and benchmarks both extractive and abstractive methods.
- Earlier fiction work: Prior fiction-summarization work generated extractive overviews of short fiction using decision trees or manually designed rules.Experiments were conducted on 23 short stories.
- Earlier fiction work: Book summarization was introduced with resources and baselines based on a curated set of 50 books.The cited work established the task but preceded BOOKSUM’s broader multi-granularity collection.
- Earlier fiction work: Ladhak et al. studied paragraph-to-summary alignment for novel chapter summarization and created a silver-standard extractive dataset.Their work also evaluated extractive models for chapter summarization.
- BOOKSUM extension: BOOKSUM prioritizes abstractive summarization, aligns paragraph-, chapter-, and full-book data, and substantially increases the number of examples.It also benchmarks state-of-the-art extractive and abstractive methods across all introduced subsets.
3 Dataset
BOOKSUM combines public-domain literary sources with independently collected summaries, manual and automatic alignment, and book-level data splits. Its statistics show long, highly abstractive examples with salient content distributed throughout sources.
- Data Collection: Sources come from Project Gutenberg’s public-domain plays, short stories, and novels, while associated book- and chapter-level summaries come from the Web Archive.The texts were downloaded in plain text and summaries were processed using BeautifulSoup.
- Data Collection: Independent source and summary collection produced multiple summaries for some texts, and manual inspection found substantial stylistic differences among related summaries.The authors considered this coverage overlap advantageous for the dataset.
- Pre-processing and alignment: The corpus was cleaned, manually chapterized after heuristic extraction, and aligned from full texts to paragraphs through automatic procedures followed by manual inspection and fixes.Paragraph-sentence alignment used SentenceTransformer similarities after a human-based study of alignment strategies.
- Data Splits: 80/10/10% train, validation, and test splits were assigned by book title to keep all examples from a book in one subset.This prevents leakage across paragraph-, chapter-, and full-book examples belonging to the same title.
- Data Statistics: 217 book titles and 6,327 chapters yield 146,532 paragraph-level, 12,630 chapter-level, and 405 book-level examples.Source and summary lengths increase substantially with granularity, including chapter inputs averaging over 5000 words.
- Data Analysis: Salient words are distributed relatively evenly across four source segments, unlike datasets where most salient words occur in the first quarter.This suggests models need to use the entire source document for high-quality summaries.
- Data Analysis: BOOKSUM summaries are highly abstractive across measured n-gram sizes and are more abstractive than most prior datasets except XSum.High trigram novelty indicates that the summaries do not contain long extractive spans.
- Qualitative Study: Summaries at different granularities can cover the same content with high semantic and low lexical overlap, providing training signal for abstractive models.Chapter- and book-level summaries partially overlap in content but differ substantially in detail.
4 Experiments
The experiments benchmark extractive and abstractive models across BOOKSUM’s paragraph-, chapter-, and book-level tasks, using automatic and human evaluation. Results show that long-form narrative summarization remains difficult for current models and evaluation metrics.
- Experimental setup: BOOKSUM baselines were evaluated on paragraph-, chapter-, and book-level alignment tasks using extractive and abstractive architectures.The setup includes standard extractive baselines, pretrained abstractive models, and a generate-and-rank procedure for longer inputs.
- Experimental setup: Generate-and-rank applies paragraph-level models to longer documents by generating paragraph summaries and ranking them by confidence.Abstractive candidates are ranked by perplexity, while extractive candidates use model-assigned scores.
- Automatic evaluation: BERTScore values were low and decreased as reference summaries lengthened, with the highest scores slightly above 0.19 for fine-tuned T5 on paragraphs.The authors suggest BERTScore may be poorly suited to highly abstractive, long summaries.
- Extractive baselines: 17.99, 14.32, and 6.50 are Lead-3’s R-1 scores on paragraph-, chapter-, and book-level data, respectively.Random Sentences closely trails Lead-3, suggesting BOOKSUM is less susceptible to layout biases than some other domains.
- Extractive models: MatchSum performs best among extractive models on paragraphs but underperforms CNN-LSTM and BertExt on chapter and book summaries.CNN-LSTM is better on paragraph data, whereas BertExt performs better on chapters and books.
- Abstractive models: Fine-tuning generally improves abstractive models across granularities; BART performs best on paragraphs and chapters, while T5 performs best on books.Exceptions include zero-shot BART on book-level ROUGE and fine-tuned T5 on SummaQA; PEGASUS underperforms other baselines.
- Human evaluation: Human ratings show that fluency and coherence generally decrease as source documents and summaries become longer.The authors identify longer fluent and coherent generation as a problem for existing neural models.
- Discussion: Independently generated and selected sentences can produce incoherent, overlapping, or low-significance summaries, motivating more tailored methods.The generate-and-rank experiments are presented as groundwork rather than a final solution for long-form summarization.
5 Conclusions
The paper introduces BOOKSUM as a three-granularity dataset collection for long-form narrative summarization and benchmarks extractive and abstractive baselines. Its analyses position the collection as a challenging resource for future summarization research.
- Contributions: BOOKSUM provides paragraph-, chapter-, and full-book annotations with increasing difficulty for long-form narrative summarization.The collection is designed to support comparison across granularities.
- Contributions: The authors quantitatively compare BOOKSUM with existing summarization corpora and report that it presents new challenges for summarization methods.The comparison supports the dataset’s role as a demanding benchmark.
- Contributions: Extractive and abstractive baseline models using pretrained architectures establish reference results for future methods.The baselines are intended to enable straightforward comparison on long-narrative summarization.
6 Limitations
The paper identifies legal, demographic, evaluation, and computational constraints affecting BOOKSUM data use and model assessment. These limitations narrow how the dataset and reported baseline results should be interpreted.
- Data Collection: Data use depends on copyright status, jurisdiction, purpose, and other factors considered under copyright law.The paper understands use of public-domain data as legally permissible and copyrighted-data use as allowed under U.S. fair use.
- Data Biases: BOOKSUM contains English-language books more than fifty years old, reflecting the society of that era and carrying associated dataset biases.The evaluated pretrained models also carry biases from their pretraining data, but the study did not stress-test these ethical biases.
- Model Evaluation: Automatic metrics including ROUGE, BERTScore, and SummaQA were not validated for very long documents and highly abstractive summaries.They might therefore fail to accurately reflect the evaluated models’ true performance.
- Computational Resources: Training and evaluating neural models on BOOKSUM’s long source documents might require substantial computational resources.
A Further Implementation Details
The experiments used author-recommended model configurations and pretrained checkpoints, with implementation resources and checkpoint details documented for reproducibility.
- Training Configuration: Models were trained for 10 epochs with a batch size of 16 using configurations recommended by the original model authors.
- Model Initialization: Many baselines initialized weights from pretrained checkpoints before fine-tuning on BOOKSUM data.
- Checkpoint Documentation: Table 4 lists the checkpoints used for each baseline and similarity-score model, together with approximate parameter counts.
B Data Alignment Process
The alignment process matches book paragraphs to summary sentences by scoring every pair and applying stable matching, with several lexical and neural similarity functions evaluated in a pilot study.
- B Data Alignment Process: Alignment first computes paragraph–summary sentence similarity scores for all pairs, then applies a stable matching algorithm to obtain final alignments.The procedure follows two steps described by Ladhak et al. (2020).
- Similarity Metric Selection: A pilot study sampled 100 paragraph–sentence pairs from the validation set to compare different embedding-based similarity methods.The sampled examples were matched using the same alignment procedure.
- Similarity Metrics: ROUGE-wtd and ROUGE-avg score lexical overlap using averaged ROUGE-{1,2,L}, with ROUGE-wtd additionally weighting tokens by inverse document frequency.
- Similarity Metrics: BERTScore measures semantic overlap by aligning tokens through maximum cosine similarity between their BERT representations.
- Similarity Metrics: Cross-Encoder jointly processes each sentence–paragraph pair with Transformer self-attention, whereas Bi-Encoder independently encodes each text and compares vectors with cosine similarity.Two Bi-Encoder checkpoints were evaluated as described in Table 4.
- Human Evaluation: Three English-speaking judges labeled each pilot alignment as related, somewhat related, or unrelated, after which related and somewhat related labels were merged by majority vote.Workers were recruited through Amazon Mechanical Turk and paid approximately 12 USD per hour.
- Human Evaluation: Table 5 reports how often each alignment method received a positive human label.
- B Data Alignment Process: Stable matching produces pairings where neither paragraph nor summary sentence prefers an alternative existing match.
C Alignment Quality
Alignment quality was assessed with semantic and lexical similarity measures. The reported scores indicate strong semantic overlap alongside comparatively low lexical overlap, consistent with highly abstractive pairings.
- C Alignment Quality: Alignment quality was evaluated using cosine similarity for semantic similarity and ROUGE-1, ROUGE-2, and ROUGE-L F-scores for lexical overlap.
- C Alignment Quality: 0.412 cosine similarity indicates strong semantic overlap between paired summary sentences and source paragraphs.
- C Alignment Quality: 17.39 R-1 lexical overlap is comparatively low, highlighting the high abstractiveness of the data.
D Data Creation Pipeline
The BookSum data creation process is visualized in Figure 3, while Table 7 presents chapter-level summaries of “Sense and Sensibility” collected from different sources.
- Figure 3 visualizes the data creation process.
- Table 7 presents chapter-level summaries of “Sense and Sensibility” collected from different sources.
- The examples focus on summaries of “Sense and Sensibility.”
F Human Evaluation UI
The paper documents interfaces used in human studies and provides example summaries generated by baseline models across paragraph-, chapter-, and book-level tasks.
- Human Evaluation UI: Human-study interfaces are shown for paragraph-level evaluation and for chapter- and book-level evaluation.
- Baseline Examples: Baseline-generated examples span paragraph-, chapter-, and book-level summaries.
- Summary Examples: Table 7 presents chapter-level summaries collected from different sources, with same-color underlining indicating high semantic and low lexical overlap.
- Baseline Examples: The examples include one-paragraph, chapter-level, and full-text summaries of “Sense and Sensibility.”