Source-linked AI summary
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Ziqiang Cao, Furu Wei, Wenjie Li, Sujian Li
TL;DR
Abstractive summarization can fabricate facts when it fuses source content, and nearly 30% of state-of-the-art s2s outputs exhibit this problem. The paper extracts fact descriptions with OpenIE and dependency parsing, then uses dual attention over the source and descriptions. On Gigaword, this reduces fake summaries by 80% and improves informativeness, although descriptions can be incomplete and generation may neglect some of them.
Problem
Nearly 30% of state-of-the-art s2s summaries suffer from fact fabrication, making faithfulness a vital prerequisite for practical abstractive summarization.
Method
The model extracts fact descriptions using OpenIE and dependency parsing, then conditions dual-attention s2s generation on the source sentence and descriptions.
Results
80% fewer fake summaries were produced on the Gigaword benchmark, with significant improvement in informativeness evaluation.
Takeaways & Limitations
Fact descriptions condense sentence meaning and help generation remain faithful while also improving informativeness.
Takeaways & Limitations
Fact descriptions may be incomplete, and generation can focus on one of multiple long descriptions, damaging informativeness.
Abstract
from arXiv · showhide
Unlike extractive summarization, abstractive summarization has to fuse different parts of the source text, which inclines to create fake facts. Our preliminary study reveals nearly 30% of the outputs from a state-of-the-art neural summarization system suffer from this problem. While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system. To avoid generating fake facts in a summary, we leverage open information extraction and dependency parse technologies to extract actual fact descriptions from the source text. The dual-attention sequence-to-sequence framework is then proposed to force the generation conditioned on both the source text and the extracted fact descriptions. Experiments on the Gigaword benchmark dataset demonstrate that our model can greatly reduce fake summaries by 80%. Notably, the fact descriptions also bring significant improvement on informativeness since they often condense the meaning of the source text.
Introduction
Abstractive sentence summarization must fuse source information, creating a risk of fabricated facts that can undermine practical usefulness. The paper addresses faithfulness by extracting source facts and conditioning generation on them alongside the source sentence.
- Motivation: Nearly 30% of state-of-the-art s2s summaries suffer from fact fabrication caused by mismatched source relations.The paper calls agreement with source facts summary faithfulness and identifies it as essential for practical abstractive summarization.
- Motivation: The paper proposes encoding existing source facts to prevent fake generation during abstractive summarization.This motivates extracting fact descriptions before generation.
- Approach: OpenIE relation triples and dependency-parser tuples are merged into short fact descriptions that represent sentence skeletons.These descriptions are incorporated as an additional input source alongside the original sentence.
- Approach: The dual-attention s2s model conditions generation on both the source sentence and extracted fact descriptions.The model is designed to push generated summaries toward the facts expressed in the source.
- Results: 80% fewer fake summaries were produced on the Gigaword benchmark, while fact descriptions also significantly improved automatic informativeness evaluation.The paper attributes the informativeness benefit to the descriptions’ condensed representation of sentence meaning.
Fact Description Extraction
The system extracts concise fact descriptions from OpenIE and dependency parsing, removes redundant descriptions, and feeds the resulting descriptions with the source sentence. These descriptions are informative but incomplete in some cases, motivating joint use with the source.
- OpenIE extraction: OpenIE represents facts as subject-predicate-object triples, which are joined into concise fact descriptions.Multiple triples may express the same fact at different granularities.
- Description selection: Redundant triples are removed when all their words are covered by another fact description, reducing duplication and computation burden.Remaining descriptions are concatenated with the separator “|||” when multiple descriptions are extracted.
- Dependency parsing: Dependency parsing supplements OpenIE when complete relation triples are unavailable by extracting subject-predicate or predicate-object tuples.About 15% of OpenIE outputs are empty on the dataset, whereas these partial tuples are almost present in each sentence.
- Description examples: Dependency-tree extraction can produce multiple descriptions, such as “taiwan share prices opened lower tuesday” and “dealers said.”The figure presents these as extracted fact descriptions from one dependency-tree example.
- Description utility: 40% more words from fact descriptions are used in summaries than words from the original sentence.The paper interprets this copy-ratio difference as evidence that fact descriptions condense sentence meaning.
- Limitations: Fact descriptions alone are insufficient because they are shorter than the actual summary in 20% of sentences, empty in 4%, and omit about one key source word on average.The model therefore retains the original source sentence as an additional input.
Fact Aware Neural Summarization
FTSum uses separate encoders and attentional processing for the source sentence and extracted fact descriptions, then combines their contexts during decoding. A gated combination outperforms simple concatenation, while training uses negative log-likelihood and standard optimization procedures.
- Encoders: FTSum encodes the source sentence and fact descriptions separately before decoding summaries from representations of both inputs.The sentence and relation sequences use BiGRU encoders, with boundary indicators resetting the relation encoder between independent fact descriptions.
- Dual-Attention Decoder: A dual-attention decoder constructs sentence and relation context vectors and merges them during generation.The model explores simple concatenation (FTSumc) and a gated weighted-sum approach (FTSumg).
- Dual-Attention Decoder: FTSumg significantly outperforms FTSumc, and its gate values reflect the relative reliability of sentence and fact descriptions.The gate network combines the two context vectors using a weighted sum.
- Dual-Attention Decoder: The decoder generates each next word from the previous output, context vector, and current decoder state through a softmax layer.The decoder uses a GRU with attentions rather than task-specific copying or coverage mechanisms.
- Learning: Training maximizes the estimated probability of the actual summary using negative log-likelihood, Adam, mini-batches, learning-rate adjustment, and gradient clipping.The implementation uses a learning rate of α = 0.001 and gradient clipping over [−5, 5].
Experiments
Experiments on the Annotated English Gigaword corpus compare FTSum with established baselines using perplexity, ROUGE, and manual faithfulness judgments. FTSum reduces disinformation while improving automatic informativeness, though multiple long fact descriptions can reduce coverage.
- Dataset and Evaluation: Experiments use the Annotated English Gigaword corpus, pairing news-article lead sentences with headlines as summaries.
- Dataset and Evaluation: The evaluation combines ROUGE-1, ROUGE-2, and ROUGE-L F1 scores with manual FAITHFUL, FAKE, and UNCLEAR labels.
- Automatic Evaluation: FTSum achieves the lowest development-set perplexity among systems with reported values and substantially higher ROUGE scores than comparison methods.
- Automatic Evaluation: 13% and 56% are FTSum's reported ROUGE-2 improvements over Feats2s and ABS+, respectively.
- Faithfulness Evaluation: 30% of attentional s2s outputs contain disinformation, compared with 6% for FTSum; nearly 90% of FTSum summaries are faithful.
- Limitations: With multiple long fact descriptions, FTSum sometimes uses only one item, preserving faithfulness but damaging informativeness; the authors leave coverage mechanisms for future work.
- Gate Analysis: FTSum's gate value decreases during training from above 0.5 to 0.415 as generation shifts toward fact descriptions.
Related Work
Prior sentence-summarization work developed rule-based, syntactic, statistical, and neural approaches, generally emphasizing informativeness. This paper identifies faithfulness as a distinct problem and introduces it as a research focus.
- Earlier Approaches: Earlier sentence-summarization methods included rule-based systems, syntactic tree pruning, and statistical machine translation.
- Neural Models: Encoder-decoder models such as ABS, RAS-Elman, and full RNN sequence-to-sequence systems achieved state-of-the-art performance on Gigaword.
- Task-Specific Modifications: Researchers also added task-specific features, including named entities, POS tags, and copying mechanisms, to adapt general sequence-to-sequence models.
- Research Gap: Previous research usually focused on improving summary informativeness.
- Research Gap: The paper presents itself as the first to explore faithfulness in abstractive summarization.
Conclusion and Future Work
The paper addresses faithfulness in abstractive summarization by conditioning generation on source sentences and extracted fact descriptions. It reports an 80% reduction in fake summaries and identifies decoder improvements and automatic faithfulness evaluation as future work.
- The paper investigates faithfulness in abstractive summarization and extracts fact descriptions using OpenIE and dependency parsing.
- The dual-attention sequence-to-sequence framework conditions generation on both the source sentence and extracted fact descriptions.
- 80% reduction in fake summaries is achieved on the Gigaword benchmark.
- Future work includes adapting copying and coverage mechanisms to the decoder and developing automatic evaluation of summary faithfulness.