Source-linked AI summary
Exploring Content Selection in Summarization of Novel Chapters
Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, Kathleen McKeown
TL;DR
Novel-chapter summarization is harder than news summarization because chapters are longer and reference summaries heavily paraphrase and generalize their source text. The paper develops extractive content-selection methods using improved alignment and finds that sentence-level stable matching with weighted similarity significantly outperforms prior alignment approaches.
Problem
Novel chapters require summarization methods that handle much longer texts and reference summaries with extensive paraphrasing and generalization.
Method
The paper creates gold extractive targets by aligning summary and chapter sentences with weighted ROUGE/METEOR metrics and stable matching, while comparing alternative alignment methods.
Results
Sentence-level stable-matched alignment and the proposed weighted similarity method outperform prior approaches on automated metrics and crowd-sourced pyramid evaluation.
Takeaways & Limitations
Improved extract construction is a promising first step toward summarizing novel chapters with extractive systems.
Takeaways & Limitations
Abstractive models produced low ROUGE with repetitions and hallucinations, while constituent- versus sentence-level extraction results differed between automated and human evaluation.
Abstract
from arXiv · showhide
We present a new summarization task, generating summaries of novel chapters using summary/chapter pairs from online study guides. This is a harder task than the news summarization task, given the chapter length as well as the extreme paraphrasing and generalization found in the summaries. We focus on extractive summarization, which requires the creation of a gold-standard set of extractive summaries. We present a new metric for aligning reference summary sentences with chapter sentences to create gold extracts and also experiment with different alignment methods. Our experiments demonstrate significant improvement over prior alignment approaches for our task as shown through automatic metrics and a crowd-sourced pyramid analysis. We make our data collection scripts available at https://github.com/manestay/novel-chapter-dataset .
1 Introduction
The paper introduces extractive summarization for novel chapters, a harder setting than news because chapters are longer and summaries heavily paraphrase. It develops improved alignment and evaluates it with multiple models and evaluation methods.
- Motivation: The task targets extractive summaries of novel chapters, which are substantially longer than news articles and contain extensive paraphrasing.Novel chapters average seven times the length of news articles, with no one-to-one correspondence between summary and chapter sentences.
- Approach: A new alignment approach weights ROUGE and METEOR to align salient words between abstractive summaries and chapter sentences.The approach uses stable matching and experiments with BERT-based alignment.
- Results: Using the new alignment method significantly improves results over prior work across automated metrics and crowd-sourced pyramid evaluation.The method is tested within three models developed for single-document news summarization.
- Granularity: The paper also evaluates extraction at different granularities, finding mixed results for constituents versus sentences.The authors hypothesize that summary sentences combine information from several chapter sentences.
- Contributions: The work contributes a new summarization task, experiments exposing problems with previous extract construction, and an improved method for gold extracts.Two abstractive models produced low ROUGE and poor outputs with repetitions and hallucinations.
2 Related Work
Prior novel summarization work is limited, while related research has explored unsupervised methods, character summaries, discourse-unit extraction, and phrase-level extraction. Fully abstractive methods remain unsuitable for this task because of extensive paraphrasing and generalization.
- Novel Summarization: Novel summarization has received relatively little attention, with earlier work providing novel-summary datasets and unsupervised systems, and later work focusing on character summaries.The cited prior systems drew on Meade and TextRank.
- Extractive Methods: Related extractive work uses Rhetorical Structure Theory to select elementary discourse units for compression and denser summaries.Other neural methods use attention to focus extraction on phrases within sentences.
- Task Fit: Fully abstractive methods are not yet appropriate for novel chapters because the summaries involve extensive paraphrasing and generalization.Semantic-textual-similarity resources also produce poorer alignments for this domain than the methods evaluated in the paper.
3 Data
The dataset pairs novel chapters from Project Gutenberg with human-written study-guide summaries and applies filtering to create train, development, and test splits. Its chapters and summaries are much longer than news data, and their lower word overlap reflects substantial paraphrasing.
- Collection: The dataset contains 4,383 unique chapters paired with two to five human-written summaries from five online study guides.The chapters are drawn from 79 books and the initial collection contains 9,560 chapter-summary pairs.
- Availability: The dataset cannot be redistributed because the authors do not have the rights, but the data-collection scripts are available online.The scripts are provided at the project repository referenced by the paper.
- Filtering: Filtering removes chapters exceeding 700 sentences and summaries with compression ratios below 2.0 because they are too large or contain substantial commentary.The first filter removes approximately 10% of the data and the second approximately 5%.
- Splits: The filtered dataset contains 8,088 pairs split into 6,288 training, 938 development, and 862 test pairs.Books, rather than individual pairs, are randomly assigned to the splits.
- Dataset Scale: Novel chapters average 5,165 words and summaries 372 words, compared with 761 and 46 words for CNN/DailyMail news.Thus chapters are seven times longer and summaries eight times longer than the corresponding news texts.
- Paraphrasing: Word overlap is 33.7% for novel summaries versus 68.7% for CNN/DailyMail summaries, indicating substantially more paraphrasing.Figure 1 illustrates summaries paraphrased from several nonconsecutive chapter sentences and varying in detail.
4 Alignment Experiments
The paper evaluates similarity metrics and alignment strategies for matching abstractive chapter summaries to chapter sentences, finding that weighted metrics and sentence-level stable matching produce better alignments.
- 4.1 Similarity Metrics: The study compares ROUGE variants, METEOR-augmented metrics, and BERT cosine similarity to accommodate lexical variation and paraphrasing.The weighted and unweighted ROUGE-plus-METEOR variants are evaluated alongside BERT representations for chapter-summary alignment.
- 4.1 Similarity Metrics: Crowd workers evaluate whether generated alignments convey each reference summary sentence, enabling precision and recall estimates for content overlap.The evaluation presents an alignment for each reference summary and asks workers to judge up to 12 summary sentences.
- 4.1 Similarity Metrics: Humans significantly prefer R-wtd alignments, which weight salient words rather than treating every word equally.R-wtd averages weighted ROUGE-1, ROUGE-2, and ROUGE-L with stemming; the weighting uses smooth inverse frequency estimates from the chapter text.
- 4.2 Alignment Methods: Stable matching applies Gale-Shapley preferences to prevent chapter and summary segments from preferring alternative pairings over their assigned matches.This sentence-level method is contrasted with the greedy approach used in prior work.
- 4.2 Alignment Methods: Human evaluation finds that sentence-level stable matching significantly outperforms summary-level alignment and is used thereafter.The comparison covers selecting chapter sentences by reference word count or ROUGE saturation versus stable matching at the sentence level.
5 Summarization Experiments
The experiments compare extractive summarization systems trained with sentence- and constituent-level targets derived from alternative alignment methods. The proposed ROUGE-weighted stable-matching alignment improves results across systems, while human evaluation favors sentence extraction.
- 5.1 Results: The study trains three extractive systems—CB, K, and N—and evaluates generated summaries with ROUGE-{1,2,L} and METEOR against 2–5 reference summaries per chapter.Target summary length is set from average lengths of similarly long training chapters.
- 5.1 Results: Constituent extraction uses the longest embedded S-tag constituent so selected fragments remain sufficiently long to be meaningful.The method responds to observations that summary sentences often combine information from multiple chapter sentences.
- 5.1 Results: ROUGE-L improves 10% to 18% relatively over baselines when systems use the proposed extractive targets.With the proposed targets, all three models perform similarly, indicating that target-generation differences rather than architecture explain the apparent baseline gap.
- 5.1 Results: Sentence-level stable matching produces significantly higher content overlap with reference summaries than the baseline alignment in crowd-sourced pyramid evaluation.The comparison evaluates CB with R-wtd for sentence and constituent extraction against CB with K-align.
6 Discussion and Conclusion
The paper concludes that sentence-level stable matching and ROUGE-weighted alignment improve novel-chapter extractive summarization. However, automated and human evaluations disagree about sentence versus constituent extraction, motivating further work on fluent constituent combinations and long-document abstractive models.
- 6 Discussion and Conclusion: Sentence-level stable-matched alignment outperforms summary-level alignment, while ROUGE-weighted matching produces better gold extracts than competing similarity metrics.The resulting system is presented as a first step toward summarizing novel chapters.
- 6 Discussion and Conclusion: Automated metrics and human evaluation agree that the new alignment approach outperforms prior approaches but disagree on constituent versus sentence extraction.The discrepancy may arise because automated scoring omits constituent context, whereas human evaluation includes context for readability.
- 6 Discussion and Conclusion: Large pretrained abstractive models remain difficult to apply because novel chapters are about an order of magnitude longer than their 512-token pretraining inputs.Their memory demands limit input length, and truncation may hurt performance in this domain.
A.2 Example Chapter and Summaries
The paper illustrates the task with Chapter 11 of The Awakening, its reference summaries, and generated summaries from the best alignment method. The examples share covered content despite differences in length and detail.
- A.2 Example Chapter and Summaries: Chapter 11, The Awakening by Kate Chopin, is shown as the example source chapter.
- A.2 Example Chapter and Summaries: Three reference summaries and two generated summaries using the best alignment method are presented for the chapter.
- A.2 Example Chapter and Summaries: The reference and generated summaries differ in length and detail but show clear similarities in covered content.
A.3 Target Word Length for Summaries
The target summary length is determined from the chapter length and an estimated compression ratio, using quantile-specific averages from the training set.
- A.3 Target Word Length for Summaries: Generated summaries use a target word length based on chapter word count and the mean compression ratio for its training-set quantile.The training set is divided into 10 quantiles, each associated with a mean compression ratio.
- A.3 Target Word Length for Summaries: The compression ratio compares chapter word count with reference-summary word count.
- A.3 Target Word Length for Summaries: Generated summaries extract the highest-probability segments until the target word-length budget is reached without truncation.Oracle summaries use the same target length but may be shorter when the reference summary contains few segments.
A.4 SCU Evaluation Task Setup
The SCU evaluation introduces distractor statements with predetermined correctness so evaluation questions can test whether generated summaries support them.
- A.4 SCU Evaluation Task Setup: The evaluation inserts one positive distractor into both the generated summary and statement list, making it correct.
- A.4 SCU Evaluation Task Setup: A negative distractor is inserted only into the statement list, making it incorrect.
- A.4 SCU Evaluation Task Setup: The two distractors are sampled from different chapters of the same book.
A.5 Constituent Extraction algorithm
The constituent extraction algorithm converts parse-tree structure into meaningful text segments by selecting clauses, verb phrases, and remaining uncovered nodes while avoiding duplicated words.
- A.5 Constituent Extraction algorithm: The algorithm extracts relative clauses, suitable S or SBAR nodes, VP nodes above them, and remaining uncovered tree nodes as constituent segments.These categories define the intended hierarchy of extracted spans.
- A.5 Constituent Extraction algorithm: It searches constituent subtrees in a parse tree and identifies nodes containing NP and VP children before ascending toward the root.
- A.5 Constituent Extraction algorithm: The procedure breaks conjunction clauses into separate segments and removes added subtrees from the original tree to avoid duplicating words.
- A.5 Constituent Extraction algorithm: The extracted word lists are concatenated into short segments and joined into the final constituent sequence c1, ..., cn.