Source-linked AI summary
Neural Extractive Text Summarization with Syntactic Compression
Jiacheng Xu, Greg Durrett
TL;DR
Neural summarization has mainly emphasized extraction or abstraction, motivating a model that combines sentence selection with syntactic compression. The model jointly learns extraction and compression decisions from oracle supervision, achieves strong ROUGE results comparable to state of the art, and generally produces grammatical output.
Problem
Neural summarization has largely separated robust extraction from flexible abstraction, while neural models combining extraction and compression have received little study.
Method
The model encodes a document, selects sentences, chooses syntax-derived discrete compression options, and jointly learns extraction and compression from oracle supervision.
Results
The model matches or exceeds state-of-the-art performance across CNN, Daily Mail, and New York Times datasets, with a +2.4 ROUGE-F1 gain over the extractive baseline on CNN.
Takeaways & Limitations
Joint extraction and syntactic compression provide strong summarization performance, outperform an off-the-shelf compression module, and generally retain grammaticality.
Takeaways & Limitations
Human grammaticality ratings were noisy, and many remaining output errors were attributed to the parser.
Abstract
from arXiv · showhide
Recent neural network approaches to summarization are largely either selection-based extraction or generation-based abstraction. In this work, we present a neural model for single-document summarization based on joint extraction and syntactic compression. Our model chooses sentences from the document, identifies possible compressions based on constituency parses, and scores those compressions with a neural model to produce the final summary. For learning, we construct oracle extractive-compressive summaries, then learn both of our components jointly with this supervision. Experimental results on the CNN/Daily Mail and New York Times datasets show that our model achieves strong performance (comparable to state-of-the-art systems) as evaluated by ROUGE. Moreover, our approach outperforms an off-the-shelf compression module, and human and manual evaluation shows that our model's output generally remains grammatical.
1 Introduction
The paper introduces a jointly trained neural model that combines sentence extraction with syntactic compression. Experiments on news summarization datasets show strong ROUGE performance and generally grammatical compressed output.
- 1 Introduction: Neural summarization has largely divided between robust extractive systems and flexible but potentially error-prone abstractive systems, leaving limited neural work combining extraction and compression.The proposed approach aims to combine extraction performance, compression flexibility, and discrete interpretability.
- 1 Introduction: The model sequentially selects document sentences and chooses syntax-derived compression options using document, sentence, and decoder context.Extraction and compression are jointly learned from supervision derived from oracle extractive-compressive summaries.
- 1 Introduction: +2.4 ROUGE-F1 on CNN over the extractive baseline, while matching or exceeding state-of-the-art systems across CNN, Daily Mail, and New York Times datasets.The authors attribute the largest CNN improvement to the more compressed nature of CNN summaries.
- 1 Introduction: Human evaluation finds the compressed output generally grammatical, with many remaining errors attributed to the parser.The evaluation also examines the robustness and tunability of the model’s compression threshold.
2 Compression in Summarization
The paper frames syntactic compression as a way to vary summary length while preserving grammaticality through discrete deletion options. It argues that ROUGE-supervised joint learning is important for selecting useful compressions for each source.
- 2 Compression in Summarization: Syntactic compression deletes selected low-criticality material to shorten sentences, while the model learns which grammatical options should actually be removed.The candidate options include appositives, clauses, adjective and adverbial phrases, selected prepositional phrases, gerundive phrases, and parentheticals.
- 2 Compression in Summarization: The compression options are syntax-derived and designed so that all combinations preserve grammaticality, although some contextually important elements should remain undeleted.The model must learn not to delete important verb-phrase and prepositional-phrase content.
- 2 Compression in Summarization: Summary sources vary from fully extractive to heavily compressed, motivating evaluation on datasets with different levels of compressibility.The paper examines compressability across summarization datasets to motivate its evaluation choices.
- 2 Compression in Summarization: ROUGE supervision is critical for learning source-specific information importance, and the proposed model substantially outperforms an off-the-shelf compression module.The paper notes that labeled compression data is unavailable in many domains.
3 Model
JECS jointly selects document sentences and applies syntactically derived compression options using neural extraction and compression modules. The model encodes document context, scores sentence selections and deletions, and postprocesses redundant compressed spans.
- 3 Model: JECS encodes the document, selects sentences, and applies discrete compression options derived from syntactic rules.Its architecture contains separate sentence-extraction and text-compression modules.
- 3.1 Extractive Sentence Selection: The extraction module represents words and sentences with BiLSTMs and CNNs, aggregates sentence representations into a document vector, and decodes sentence selections sequentially.The decoder conditions on the last selected sentence, document representation, and recurrent state while excluding already selected sentences.
- 3 Model: At test time, greedy decoding selects sentences, while fixed dataset-specific sentence counts and postprocessing remove compression options whose unigrams are redundant elsewhere.The fixed number of sentences is tuned for each dataset, and deduplication is applied after prediction.
- 3.2 Text Compression: Compression candidates are spans identified by syntactic rules, including appositives, clauses, modifiers, selected prepositional phrases, and parentheticals.Each candidate is represented as a possible deletion option for a selected sentence.
- 3.2 Text Compression: A neural compression classifier combines sentence, candidate-span, decoder, and document representations to predict whether each candidate should be deleted or kept.The classifier treats each compression decision as binary classification.
4 Training
Training constructs extractive-compressive oracle summaries and jointly learns sentence selection and compression decisions from them.
- 4.1 Oracle Construction: Oracle construction first identifies sentence combinations with beam search, then derives compression labels for the selected sentences.The extractive oracle is reused for both the extraction-only and joint models.
- 4.1 Oracle Construction: The oracle procedure assigns one compression label per option independently of the context in which the option occurs.This simplification avoids the prohibitive cost of recomputing context-dependent decisions with a dynamic oracle.
- 4.1 Oracle Construction: CNN is the most compressible dataset, with substantially more ROUGE-improving compression options than the other datasets.Table 2 separates negative, weak-positive, and strong-positive compression labels by dataset.
- 4.1 Oracle Construction: Compression options are labeled positive when deleting a phrase improves ROUGE, using ROUGE before and after compression.Options that decrease ROUGE receive negative labels, while ROUGE-improving options are applied.
- 4.2 Learning Objective: Training averages sentence-selection log likelihood across five similarly scoring oracle summaries instead of committing to one.Oracle sentence indices are ordered by individual salience rather than document order.
- 4.2 Learning Objective: The model jointly learns extraction and compression with a combined loss, L = Lsent + αLcomp, using α = 1.The two components are supervised together rather than optimized independently.
5 Experiments
Experiments show that joint extraction and compression improves summarization quality across CNN/Daily Mail and NYT while generally preserving grammaticality. The model outperforms extractive, lead-based, pipeline, and compression-augmented baselines, though compression errors remain.
- 5.1 Experimental Setup: Compression yields larger gains on CNN because its reference summaries are shorter and more compressed than those of Daily Mail.The experiments therefore focus first on CNN before evaluating the other datasets.
- 5.2 Results on CNN: +2 ROUGE F1 over baselines and prior systems on CNN, with joint JECS outperforming compression in isolation and the EXTLSTMDEL pipeline.The result supports jointly training extraction and compression with ROUGE supervision.
- 5.3 Results on Combined CNNDM and NYT: JECS improves over extractive and lead-based systems across CNNDM and NYT, although it slightly underperforms on ROUGE-2 in some cases.The authors attribute one possible source of ROUGE-2 weakness to stop-word removal during oracle construction.
- 5.4 Grammaticality: Compared with the off-the-shelf EXTLSTMDEL compressor, JECS achieves substantially higher ROUGE while receiving roughly equal human grammaticality preference.The comparison uses matched compression ratios.
- 5.4 Grammaticality: Automatic checks find JECS makes only a small number of grammar errors, but manual analysis identifies parsing, adjective-deletion, attachment, and construction-handling errors.The manual analysis examined 40 model summaries, while Grammarly evaluated 500 CNN sentences.
6 Compression Analysis
The model’s compression threshold is robust across settings while controlling summary compression, and its selected compressions include syntactic constituents and deduplicated material.
- The model outperforms the extractive baseline across a range of compression thresholds.
- 0.45 gives the best average ROUGE, while performance remains strong from 0.3 to 0.55.The threshold controls how much content is deleted from summaries.
- Temporal and location prepositional phrases are often compressed because they can be redundant across sentences.
- Table 8 reports compression types, average lengths, oracle agreement, and the share arising from deduplication versus model prediction.
- 80% compression accuracy against ground truth is achieved without manual deduplication, although some errors have little effect on ROUGE.
7 Related Work
Related work spans neural extraction, syntactic compression, and neural sentence compression, while this approach emphasizes explicit syntax and integrated extraction with compression.
- Prior extractive summarizers formulate sentence selection as classification, ranking, or sequence-to-sequence decoding.
- Earlier syntactic compression systems used constituency parses and discourse representations, establishing precedents for syntax-driven summarization.
- Neural compression includes deletion-based LSTMs, deep generative models, and compression modules paired with extraction.
- The authors argue that explicit syntax provides more understandable and controllable compression options.
- A contemporaneous extractive-compressive model integrated compression into sequential decoding but did not leverage explicit syntax.
8 Conclusion
The paper presents a jointly trained neural framework that combines sentence extraction with syntax-derived compression and reports strong ROUGE performance with generally acceptable grammaticality.
- The model combines sentence extraction with a classifier that deletes syntax-derived compression options.
- Oracle extraction and compression decisions are learned using beam search and heuristics.
- The model outperforms prior work on CNN/Daily Mail, improves substantially over extraction alone, and produces generally acceptable grammatical output.
A Experimental Setup
The experiments use preprocessed CNN/Daily Mail and New York Times data with established filtering and implementation settings, alongside a human-evaluation interface.
- CNN/Daily Mail uses the non-anonymized version, while New York Times examples with abstracts shorter than 50 words are filtered out.
- The model uses pretrained embeddings, 200-dimensional sentence and document representations, ELMo for compression, dropout, and Adam optimization.
- Amazon Mechanical Turk provides the interface for human evaluation.
C Type Analysis
Compression options are analyzed across the CNN dataset, where PP attachments and adjectives dominate the available choices. Oracle statistics indicate that many such constituents can be compressed without hurting ROUGE, particularly in CNN's short-reference-summary setting.
- C Type Analysis: PP attachments and adjectives are the two most common compression options in CNN.Table 10 reports these constituency types as the top two compression options.
- C Type Analysis: Compression is most effective on CNN, which has the shortest reference summaries among the listed datasets.The dataset statistics identify CNN as the shortest-reference-summary setting and the strongest compression setting.
- C Type Analysis: More than half of PP constituents and almost all adjectives are compressible without hurting ROUGE according to the oracle.The passage reports oracle-supported compressibility for these two constituent types.