Source-linked AI summary
GSum: A General Framework for Guided Neural Abstractive Summarization
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, Graham Neubig
TL;DR
Neural abstractive summarization is flexible but can be unfaithful and hard to control, while existing guidance strategies are difficult to compare. GSum provides one framework for diverse external guidance and achieves state-of-the-art ROUGE on four benchmarks with highlighted sentences, while producing more faithful and controllable summaries.
Problem
Existing abstractive summarizers can produce unfaithful, difficult-to-control outputs, and prior guidance methods leave comparisons and complementarity across guidance types unclear.
Method
GSum modifies neural encoder-decoder summarizers to use diverse external guidance, including highlighted sentences, with oracle guidance during training and automatic or user-specified guidance at test time.
Results
GSum achieves state-of-the-art ROUGE on 4 of 6 benchmarks with highlighted-sentence guidance, including 1.28/0.79/1.13 ROUGE-1/2/L improvements on CNN/DM.
Takeaways & Limitations
Different guidance types yield qualitatively different summaries, and the guided models generate more faithful summaries and more novel words.
Takeaways & Limitations
The framework leaves future work in ensembling guidance-specific models, adding copy or coverage mechanisms, and testing other guidance signals.
Abstract
from arXiv · showhide
Neural abstractive summarization models are flexible and can produce coherent summaries, but they are sometimes unfaithful and can be difficult to control. While previous studies attempt to provide different types of guidance to control the output and increase faithfulness, it is not clear how these strategies compare and contrast to each other. In this paper, we propose a general and extensible guided summarization framework (GSum) that can effectively take different kinds of external guidance as input, and we perform experiments across several different varieties. Experiments demonstrate that this model is effective, achieving state-of-the-art performance according to ROUGE on 4 popular summarization datasets when using highlighted sentences as guidance. In addition, we show that our guided model can generate more faithful summaries and demonstrate how different types of guidance generate qualitatively different summaries, lending a degree of controllability to the learned models.
1 Introduction
Abstractive summarization offers fluent, flexible generation but can produce unfaithful content and is difficult to control. GSum addresses these problems with a general framework for using diverse guidance signals.
- Abstractive methods generate novel words and sentences, but their unconstrained outputs can contain factual errors and hallucinated content.
- Guided summarization provides signals intended to reduce deviation from the source and enable user-specified control over summary content.
- Previous guidance methods target individual signals such as length, keywords, or retrieved summaries, leaving their relative and complementary value unclear.
- GSum is a general, extensible framework that accepts different kinds of external guidance and modifies neural encoder-decoder models to attend to source documents and guidance.
- GSum achieves state-of-the-art ROUGE performance on 4 of 6 benchmarks with highlighted-sentence guidance, including 1.28/0.79/1.13 ROUGE-1/2/L improvements on CNN/DM.
- Different guidance signals can produce qualitatively different summaries, while analyses report greater faithfulness and more novel words.
2 Background and Related Work
Summarization research distinguishes extractive selection from abstractive generation and has explored guidance through multiple input forms. GSum builds on this varied literature by accommodating tokens, triples, sentences, and summaries within one framework.
- Neural summarization models are trained by encoding a source document and maximizing the conditional likelihood of target summary words.
- Guided summarization adds a signal g alongside source document x, conditioning generation on both inputs.
- Prior neural approaches guide generation with keywords, retrieved summaries, relational triples, desired lengths, styles, or highlighted source sentences.
3 Methods
GSum encodes source documents and guidance separately, then lets the decoder attend to both while supporting automatic or oracle guidance construction. The framework experiments with highlighted sentences, keywords, relations, and retrieved summaries.
- 3.1 Model Architecture: The model feeds source documents and guidance signals into separate encoders, with a decoder attending to both representations.
- 3.2 Choices of Guidance Signals: The framework experiments with highlighted sentences, keywords, relations, and retrieved summaries, using task-specific extraction or prediction procedures.
- 3.1.1 Encoder: Shared parameters in the lower encoder layers reduce computation and memory, while higher layers are intended to capture differences between sources and guidance.
- 3.1.2 Decoder: After self-attention, the decoder attends first to guidance and then to the full source document so guidance can direct focus while source context supplies details.
- 3.2 Guidance Construction: At test time, guidance can be manually specified or automatically predicted from the input; training uses automatic prediction or oracle extraction based on source and target.
- 3.2 Guidance Construction: Oracle guidance selects highly informative signals using reference summaries, whereas automatic guidance is predicted or retrieved using only test-time source information.
4 Experiments
Experiments evaluate GSum across six summarization benchmarks and multiple guidance signals, comparing automatic and oracle guidance with baseline and state-of-the-art systems. Highlighted-sentence guidance delivers the strongest broad gains, while results vary with dataset extractiveness and guidance predictability.
- Datasets: GSum is evaluated on six summarization benchmarks, including abstractive, extractive, news, biomedical, and knowledge-base datasets.The experiments include Reddit, XSum, CNN/DM, WikiHow, NYT, and PubMed.
- Guidance Signals: Highlighted sentences and keywords substantially outperform the abstractive baseline on CNN/DM, with highlighted sentences exceeding the best baseline by more than 1 ROUGE-L point.Relations and retrieved summaries do not improve baseline performance when automatically predicted.
- Guidance Signals: Oracle guidance improves every guidance variety, with the best-performing model reaching a ROUGE-1 score of 55.18.These results suggest that better guidance prediction could further improve performance.
- Other Datasets: Guided models perform better on more extractive datasets, whereas they do not improve strong abstractive baselines on Reddit and XSum.The highlighted-sentence model improves over baselines on extractive datasets such as PubMed and NYT.
- Analysis: Guided summaries contain more novel n-grams and are judged more faithful to their source documents than baseline summaries.A human evaluation sampled 100 test examples and used three raters scoring factual correctness from 1 to 3.
A Greedy Selection Algorithm
The algorithm uses an oracle to greedily select source sentences with the highest ROUGE scores against the reference summary, and applies the same strategy to relation triples.
- The oracle greedily selects source sentences with the highest ROUGE scores relative to the reference summary.
- Relation triples are flattened into concatenated text and treated as source sentences before applying the same greedy selection algorithm.
B Analysis
The analysis section focuses on CNN/DM, using oracle extractions during training and BertAbs as the base model unless stated otherwise.
- The analysis is conducted on CNN/DM unless otherwise stated.
- Training uses oracle extractions, with BertAbs serving as the base model unless otherwise stated.
B.1 Controllability
The controllability analysis examines how guidance quality affects output quality and how compatible or user-specified guidance changes generated summaries. Stronger guidance–output correlation and degraded performance from incompatible inputs support controllability.
- Randomly sampled guidance signals are used to plot the relationship between guidance quality and output quality.
- A strong correlation between guidance quality and output quality indicates that the model's outputs are controllable through guidance.
- Incompatible guidance signals can degrade performance, further demonstrating control through user-specified inputs.
B.2 Semantic Similarity
The semantic similarity analysis evaluates guided model outputs against references using METEOR in exact-match and full modes. Guided models outperform BertAbs, while BertExt performs best, possibly because METEOR favors longer summaries.
- METEOR evaluates semantic similarity in exact-match and full modes, rewarding exact matches or matches involving stems, synonyms, and paraphrases.
- All guided models outperform BertAbs on both METEOR measures.
- BertExt achieves the best METEOR performance, possibly because the metric favors long summaries.
B.3 Automatic Factual Correctness Evaluation
The authors find that factCC assigns the gold reference about 10% accuracy, making its score unsuitable as a standalone indicator of factual errors in generated summaries.
- Because the model is optimized toward the gold reference, factCC may not reliably indicate factual errors in generated summaries.
- The paper therefore supplements automatic factual-correctness evaluation with human evaluation.
- About 10% accuracy is assigned by factCC to the gold reference.
B.4 Necessity of Using Oracles During Training
The authors report that oracle selection of guidance signals during training is necessary beyond highlighted-sentence models, because weakening guidance–reference relevance substantially harms performance.
- Oracle selection of guidance signals during training is necessary for highlighted-sentence models.
- Significantly worse performance occurs for the other guidance signals without this methodology.
- The results further support the hypothesis that weakened guidance–reference relevance prevents models from learning to depend on guidance signals.
B.5 Domain Adaptation.
Under domain adaptation, guided abstractive models outperform abstractive baselines but do not surpass extractive baselines, indicating incomplete dependence on guidance signals across domains.
- B.5 Domain Adaptation: Domain adaptation trains a summarization model on one dataset and tests it on other datasets.
- B.5 Domain Adaptation: Extractive models generally outperform abstractive models under domain adaptation settings.
- B.5 Domain Adaptation: The guided model achieves better performance than abstractive baselines but cannot outperform extractive baselines despite receiving extracted sentences.
- B.5 Domain Adaptation: The authors suggest occasionally dropping input documents during training so the model learns to condition more strongly on guidance.
- B.5 Domain Adaptation: Table 14 reports sentence-guided-model performance across source and target domains, with the first and second rows representing those domains respectively.