PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward

Luyang Huang, Lingfei Wu, Lu Wang

arXiv:2005.01159v1cs.CL

TL;DR

Abstractive summarizers often generate fabricated or near-extractive content, motivating structured semantic representations. ASGARD combines sequential and graph encoders with a multiple-choice cloze reward, and reports stronger graph-augmented performance plus more informative, less unfaithful outputs. Its evaluation also shows that ROUGE correlates weakly with summary errors.

  • Problem

    Existing abstractive summarization models commonly produce fabricated content and near-extractive summaries, reflecting limited semantic interpretation of inputs.

  • Method

    ASGARD combines a sequential document encoder, graph-structured encoder, and multiple-choice cloze reward to model global context, local entity characteristics, and entity interactions.

  • Results

    Human evaluations find graph-augmented models trained with the cloze reward more informative and significantly less unfaithful, while ASGARD also outperforms graph-free variants in ROUGE.

  • Takeaways & Limitations

    ASGARD supports knowledge graph augmentation and semantic-driven cloze rewards as effective components for improving abstractive summary quality.

  • Takeaways & Limitations

    ROUGE correlates weakly with the three evaluated summary-error types, motivating new metrics for gauging summary quality.

Abstract

from arXiv · show

Sequence-to-sequence models for abstractive summarization have been studied extensively, yet the generated summaries commonly suffer from fabricated content, and are often found to be near-extractive. We argue that, to address these issues, the summarizer should acquire semantic interpretation over input, e.g., via structured representation, to allow the generation of more informative summaries. In this paper, we present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD. We propose the use of dual encoders---a sequential document encoder and a graph-structured encoder---to maintain the global context and local characteristics of entities, complementing each other. We further design a reward based on a multiple choice cloze test to drive the model to better capture entity interactions. Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets. We also obtain better or comparable performance compared to systems that are fine-tuned from large pretrained language models. Human judges further rate our model outputs as more informative and containing fewer unfaithful errors.

1 Introduction

ASGARD addresses unfaithful and near-extractive abstractive summaries by combining document and graph encoders with a semantic-driven cloze reward. Its graph augmentation captures entity interactions and improves reported automatic and human-evaluated summary quality.

  • Existing abstractive summarization models frequently produce unfaithful content and near-extractive summaries, indicating limited semantic interpretation of inputs.
  • ASGARD augments a document encoder with a graph-structured encoder built from OpenIE outputs to preserve global context and local entity characteristics.
  • Its graph variants capture long-range entity interactions, paragraph-level interactions, and topic shifts while reducing redundancy during summary generation.
  • A multi-choice cloze reward removes connected or co-occurring entity pairs from questions to drive semantic learning of entity interactions with reinforcement learning.
  • ASGARD models achieve significantly better ROUGE scores than graph-free variants on New York Times and CNN/Daily Mail datasets.
  • Human judges rate graph-augmented outputs as more informative and less unfaithful than outputs from models without graph encoders.

2 Related Work

Prior work applies graphs and reinforcement learning to summarization, but often overlooks entity interactions and relies on rewards that inadequately capture context or paraphrased content. ASGARD combines document encoding with graphs and introduces a semantic multi-choice cloze reward.

  • Graph-Augmented Summarization and Generation: Earlier graph-based summarization methods support sentence selection or token-level entity modeling, but largely ignore interactions among entities.
  • Graph-Augmented Summarization and Generation: Graph-to-sequence systems generate text from graph structures, whereas ASGARD uses graphs alongside a document encoder to provide complementary summarization information.
  • Reinforcement Learning and QA Reward: Reinforcement-learning approaches optimize informativeness with ROUGE-based rewards, although ROUGE does not always distinguish good summaries from bad ones.
  • Reinforcement Learning and QA Reward: Earlier QA rewards mainly remove single entities or words and may penalize paraphrased answers through exact-match scoring.
  • Reinforcement Learning and QA Reward: ASGARD instead uses multiple-choice cloze questions that encode entity interactions and accommodate paraphrased answers through QA-based semantic assessment.

3 Knowledge Graph Construction

ASGARD constructs document knowledge graphs from coreference resolution and OpenIE outputs, representing entities as connected nodes and predicates as edge attributes. This structure localizes salient information and connects entities distributed across the document.

  • The construction pipeline applies coreference resolution and OpenIE, then extracts subject-predicate-object triples while removing triples with arguments longer than 10 words.
  • ASGARD uses document-level graph encoding so the decoder attends to both the input document and the constructed graph.
  • Subjects and objects become graph nodes, predicates become directed-edge attributes, and coreferential mentions are collapsed into single entity nodes.
  • The resulting graph localizes salient content around entities and connects entities that are spread across the document through graph paths.

4 Summarization Model

The model combines separate document and graph encoders, allowing the decoder to attend to sequential text and structured entity information while modeling global context, local characteristics, and topic shifts.

  • 4.1 Encoders: The framework separately encodes the token sequence and knowledge graph, then generates summaries by attending to both representations.DOCGRAPH captures global entity context, while SEGGRAPH additionally models topic shifts across paragraphs.
  • 4.1 Encoders: The graph represents predicate-centered relations with directed subject-to-predicate and predicate-to-object edges, plus reverse edges and self-loops.Nodes are initialized from contextual token embeddings, with mention counts added to represent entity salience.
  • 4.1 Encoders: DOCGRAPH merges entity mentions across the document to encode global context and interactions connected by graph paths.This representation is intended to capture interactions among entities that may be far apart in the document.
  • 4.1 Encoders: SEGGRAPH encodes each paragraph as a subgraph, applies graph encoding and max-pooling, then connects paragraph representations with a BiLSTM to model topic shifts.Nodes referring to the same entity across subgraphs share an embedding.
  • 4.2 Summary Decoder: The decoder recurrently attends to graph and document contexts, concatenating both context vectors with its hidden state to produce vocabulary probabilities.The model also includes weight sharing between input embeddings and the output matrix, plus a copy mechanism.
  • 4.2 Summary Decoder: Hierarchical attention redistributes node attention using paragraph-level representations so that nodes in important paragraphs receive greater attention.This modification leverages SEGGRAPH’s modeling of topic transitions and recurrences to identify salient content.
  • 4.3 Training Objectives: Maximum-likelihood training combines sequence prediction with a node-salience masking objective.The masking layer predicts each node’s salience and gates its representation before graph encoding.

5 Reinforcement Learning with Cloze

ASGARD adds a multiple-choice cloze reward to reinforcement learning, using questions derived from reference summaries and salient context to encourage semantic awareness of entity interactions.

  • 5 Reinforcement Learning with Cloze: A second-stage reinforcement-learning objective follows maximum-likelihood training and uses a multiple-choice cloze reward to promote more faithful and informative summaries.Training uses self-critical policy gradients comparing a sampled summary with a greedily decoded baseline.
  • 5 Reinforcement Learning with Cloze: The reward combines ROUGE-1, ROUGE-2, and ROUGE-L F1 scores with a weighted cloze score.The combined reward is R(y) = Rrouge(y) + γclozeRcloze.
  • 5 Reinforcement Learning with Cloze: The cloze reward treats a generated summary as context and measures the average probability assigned by a separately trained QA model to correct answers from reference-derived questions.The design tests whether the generated summary shares salient information with the human reference.
  • 5 Reinforcement Learning with Cloze: Figure 3 illustrates constructing cloze questions and candidate answers from a reference summary and salient context, with arguments and predicates color-coded and italicized.The example uses the reference statement “Federal Reserve increases interest rates” and related salient context.
  • 5 Reinforcement Learning with Cloze: Questions are built from OpenIE triples in salient context by masking connected argument pairs or predicates and constructing distractor answers from alternative arguments or predicates.If reference summaries lack suitable triples, additional questions mask co-occurring entity pairs.

6 Experimental Setups

Experiments evaluate ASGARD on NYT and CNN/Daily Mail using automatic metrics, QA data, graph variants, reward variants, and comparisons with established systems.

  • Datasets: Experiments use the New York Times and CNN/Daily Mail summarization datasets, following prior preprocessing and experimental setups.NYT contains 588,909 training, 32,716 validation, and 32,703 test samples; CNN/Daily Mail contains 287,188, 13,367, and 11,490.
  • Datasets: The cloze QA models are trained from 1,414,336 NYT and 1,361,175 CNN/Daily Mail question-answer pairs, achieving 97% and 95% accuracy respectively.Each dataset reserves 20,000 samples for validation and 20,000 for testing.
  • Evaluation: ASGARD-SEG+Rrouge+Rcloze yields significantly higher ROUGE scores than the other ASGARD models on NYT, with approximate randomization p < 0.0005.The evaluation reports best overall results in boldface and best ASGARD results in italics.
  • Models and baselines: ASGARD compares document-level and paragraph-segment graph variants, maximum-likelihood, ROUGE, and cloze-reward training, plus NOGRAPH without the graph encoder.The model comparisons also include extractive, abstractive, reinforcement-learning, and pretrained-language-model baselines.
  • Evaluation: Both ASGARD-SEG+Rrouge+Rcloze and ASGARD-DOC+Rrouge+Rcloze obtain significantly better ROUGE scores than other model variants on CNN/Daily Mail, with p < 0.0005.The comparison includes models fine-tuned from BERT and a unified pretrained language model.

7 Results

ASGARD improves automatic, QA-based, and human evaluations over graph-free variants, while error analysis shows that standard metrics weakly reflect summary faithfulness.

  • Automatic Evaluation: ASGARD-SEG achieves ROUGE scores better than all comparisons except fine-tuned BART on NYT, with ROUGE-L comparable to BART.ASGARD-SEG’s result indicates the effectiveness of the graph-augmented framework.
  • Automatic Evaluation: Both ASGARD variants significantly outperform NOGRAPH in ROUGE, and adding the cloze reward significantly improves scores over ROUGE-only training.ASGARD-SEG also outperforms ASGARD-DOC, indicating better capture of topic drift across multiple paragraphs.
  • Automatic Evaluation: ASGARD-DOC with combined ROUGE and cloze rewards outperforms BERTSUMEXTABS and UNILM on CNN/Daily Mail and is comparable to fine-tuned BART.These systems are fine-tuned from large pretrained language models.
  • Automatic Evaluation: Graph-encoder models achieve higher QA prediction probability and accuracy on the multiple-choice cloze test than the graph-free variant.Higher cloze-test values indicate better summaries.
  • Human Evaluation: Human judges rate ASGARD-SEG as more informative and fluent than the graph-free variant, while its error pattern is comparable to human summaries.It produces significantly fewer out-of-context and deletion or substitution errors than the model without graph information.
  • Automatic Metrics and Summary Errors: Summaries with out-of-context or deletion/substitution errors receive lower cloze and ROUGE scores, but hallucination errors do not show the same pattern.Hallucination-heavy summaries can even receive slightly higher cloze and ROUGE scores.
  • Automatic Metrics and Summary Errors: ROUGE correlates weakly with the three error types, with Pearson’s r ranging from −0.19 to 0.14.The authors therefore suggest designing new metrics to better gauge summary quality.

8 Conclusion

The paper presents a knowledge graph-augmented abstractive summarization framework with a multiple choice cloze reward. The graph and reward improve entity modeling, summary content, informativeness, and faithfulness.

  • ASGARD combines knowledge graph augmentation with a multiple choice cloze reward for reinforcement learning.
  • The framework captures both local entity characteristics and global entity interactions from the input.
  • The cloze reward further improves summary content alongside the graph representation.
  • Human evaluation finds that graph-augmented models trained with the cloze reward produce more informative summaries and significantly fewer unfaithful errors.

A.1 Experiment Details

The experiments construct knowledge graphs for NYT and CNN/Daily Mail, use standard optimization settings, and incorporate pretrained BERT and RoBERTa components.

  • Knowledge graphs are constructed for both datasets, with abundant relations for each article.CNN/Daily Mail has more arguments but fewer predicates than NYT, indicating fewer coreferred entities.
  • All models use Adam with gradient clipping of 2.0 and a batch size of 32.The learning rate is 0.001 during maximum-likelihood training and 0.0001 during reinforcement learning.
  • The two datasets have comparable graph size.
  • The experiments use base BERT for candidate-answer selection and fine-tune base RoBERTa to build the question-answer model.

A.2 Human Evaluation Guideline

Human evaluation measures informativeness, fluency, and unfaithfulness in summaries produced for news articles. Annotators rate the first two aspects and label whether summaries contain defined unfaithful errors.

  • Each annotator evaluates four summaries for 100 news articles in random order.
  • Informativeness and fluency are rated on a 1-to-5 scale.Informativeness concerns necessary content coverage, while fluency concerns grammatical readability.
  • Annotators label whether each summary contains any of three types of unfaithful errors.
  • Faithfulness is defined as whether a summary accords with facts expressed in the source.
Loading 2005.01159v1…