PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Reward Reinforced Summarization with Saliency and Entailment

Ramakanth Pasunuru, Mohit Bansal

arXiv:1804.06451v2cs.CLcs.AIcs.LG

TL;DR

Abstractive summarization must preserve salient, entailed information while avoiding redundancy. This paper adds saliency- and entailment-sensitive rewards to a reinforcement-learning summarizer and combines them through alternate mini-batch optimization. The combined method achieves state-of-the-art CNN/Daily Mail results and strong test-only transfer improvements on DUC-2002.

  • Problem

    Abstractive summarization needs to preserve saliency and logical entailment while avoiding redundancy, but existing coverage-based models leave saliency and entailment insufficiently addressed.

  • Method

    The paper trains a pointer-coverage abstractive summarizer with REINFORCE using ROUGESal and length-normalized Entail rewards, combined with ROUGE through alternate mini-batch optimization.

  • Results

    The ROUGESal+Entail multi-reward model achieves the best overall results, statistically significantly outperforming baseline and ROUGE-reward models across metrics and setting a new state-of-the-art.

  • Takeaways & Limitations

    The approach achieves state-of-the-art CNN/Daily Mail performance, including human evaluation, and strong test-only improvements when transferred to DUC-2002.

  • Takeaways & Limitations

    The entailment evaluation uses the ground-truth summary as the premise because full-document premises performed worse, likely because classifiers were not trained on such long premises.

Abstract

from arXiv · show

Abstractive text summarization is the task of compressing and rewriting a long document into a short summary while maintaining saliency, directed logical entailment, and non-redundancy. In this work, we address these three important aspects of a good summary via a reinforcement learning approach with two novel reward functions: ROUGESal and Entail, on top of a coverage-based baseline. The ROUGESal reward modifies the ROUGE metric by up-weighting the salient phrases/words detected via a keyphrase classifier. The Entail reward gives high (length-normalized) scores to logically-entailed summaries using an entailment classifier. Further, we show superior performance improvement when these rewards are combined with traditional metric (ROUGE) based rewards, via our novel and effective multi-reward approach of optimizing multiple rewards simultaneously in alternate mini-batches. Our method achieves the new state-of-the-art results (including human evaluation) on the CNN/Daily Mail dataset as well as strong improvements in a test-only transfer setup on DUC-2002.

1 Introduction

Abstractive summarization must produce concise rewrites that preserve salient, entailed information while avoiding redundancy. The paper addresses these properties with ROUGESal and Entail rewards combined through multi-reward reinforcement learning.

  • Abstractive summarization rewrites long documents into short summaries, unlike extractive methods that select existing sentences or grammatical subsentences.
  • Good summaries should select salient information, follow logically from the source, and avoid redundancy.Coverage-based models address redundancy, while saliency and logical entailment remain areas for improvement.
  • ROUGESal up-weights salient summary words using saliency scores, whereas Entail rewards summaries whose sentences logically follow from the ground-truth summary.Entail also uses length normalization to avoid misleadingly high scores for very short sentences.
  • Policy-gradient models using the new rewards perform significantly better than a cross-entropy pointer-coverage baseline.
  • Combining rewards in alternate mini-batches further improves performance and avoids complex reward-scaling and weighting issues.
  • The combined approach achieves state-of-the-art results on CNN/Daily Mail, including human evaluation, and strong improvements in test-only DUC-2002 transfer.

2 Related Work

Related work spans extractive, compression-based, graph-based, discourse-based, and abstractive summarization approaches, alongside textual-entailment methods and learned saliency signals.

  • Earlier summarization research used extraction, compression, graph-based, and discourse tree-based approaches.
  • Recent summarization work shifted toward abstractive rewriting using parse trees, Abstract Meaning Representations, and neural pointer-copy and coverage models.
  • Recognizing Textual Entailment classifies sentence pairs as entailment, contradiction, or neutral and has been applied to question answering and information extraction.
  • Prior summarization studies used textual entailment to model graph-based relationships and select less redundant sentences.
  • The paper uses human-annotated SQuAD answer spans as a proxy for salient keyphrase-style information in summarization.The answer spans average 3.2 tokens.

3 Models

The model combines a pointer-copy coverage sequence-to-sequence generator with REINFORCE policy gradients, mixed cross-entropy training, and alternate optimization for multiple rewards.

  • 3.1 Baseline Sequence-to-Sequence Model: The baseline is a single-layer bidirectional-encoder, unidirectional-decoder LSTM with attention, pointer-copy, and coverage mechanisms.
  • 3.2 Policy Gradient Reinforce: REINFORCE addresses exposure bias and directly optimizes nondifferentiable evaluation metrics by sampling word sequences from the model policy.
  • 3.2 Policy Gradient Reinforce: The reinforcement objective minimizes the negative expected reward over sampled word sequences.The policy is parameterized by θ and predicts the next word while updating decoder states.
  • 3.2 Policy Gradient Reinforce: The method uses a self-critical baseline whose reward comes from the model’s test-time arg-max inference output.
  • 3.2 Policy Gradient Reinforce: Joint cross-entropy and reinforcement loss optimizes evaluation rewards while maintaining readability.The mixed loss is LMixed = γLRL + (1 −γ)LXE.
  • 3.3 Multi-Reward Optimization: Multiple rewards share model parameters but use separate optimization functions trained in alternate mini-batches.This avoids finding complex scaling and weight balances for a weighted reward combination.

4 Rewards

The paper uses ROUGE-L as a phrase-matching reward and introduces ROUGESal and Entail to incorporate saliency and logical entailment into summarization training.

  • ROUGE Reward: ROUGE-L is the basic reward because it improves all metric scores more effectively than ROUGE-1 or ROUGE-2.The paper uses “ROUGE” to mean ROUGE-L for its ROUGE-reward models.
  • ROUGE Reward: ROUGE-based rewards rely on phrase matching and n-gram overlap but do not represent salient phrase inclusion or directed logical entailment.
  • Saliency Reward: ROUGESal assigns higher weight to important salient words and phrases when calculating the ROUGE score.Its saliency weights come from a predictor trained on SQuAD sentence and answer-span pairs.
  • Entailment Reward: The Entail reward uses an entailment scorer and a multi-sentence, length-normalized extension to favor summaries logically entailed by the source document.
  • Entailment Reward: Length normalization prevents very short sentences from receiving misleadingly high entailment scores.The normalization compares generated-summary length with reference-summary length.

5 Experimental Setup

The experiments use CNN/Daily Mail for summarization, DUC-2002 for test-only transfer, and SNLI, Multi-NLI, and SQuAD to train auxiliary classifiers. Evaluation combines ROUGE and METEOR with pairwise human judgments of relevance and readability.

  • CNN/Daily Mail supplies news articles and summaries, while DUC-2002 supports test-only generalization experiments.
  • SNLI and Multi-NLI train the entailment classifier, and SQuAD trains the saliency prediction model.
  • The study reports ROUGE full-length F1 scores and METEOR scores for its summarization models.
  • Human evaluation uses Amazon Mechanical Turk to assess summary relevance and readability through pairwise model comparisons.
  • For entailment scoring, the ground-truth summary is used as a shorter premise because it is correctly entailed by the source document.

6 Results

Reinforcement-learning rewards improve over the cross-entropy pointer-coverage baseline, with ROUGESal, Entail, and especially their multi-reward combination producing stronger results. The combined model also improves human-rated quality on CNN/Daily Mail and transfers better to DUC-2002.

  • ROUGE-L reinforcement learning improves all reported metrics over the cross-entropy baseline with statistical significance (p < 0.001).The comparison also reaches statistical significance relative to See et al. (2017).
  • ROUGESal significantly improves all metrics over both the baseline and ROUGE-reward results (p < 0.001).The result supports the benefit of incorporating saliency knowledge into the summarization model.
  • ROUGE+Entail significantly outperforms ROUGE-reward on ROUGE-1, ROUGE-L, and METEOR and beats Entail-reward on all ROUGE metrics (p < 0.001).
  • ROUGESal+Entail achieves the best overall results and establishes the new state-of-the-art, significantly outperforming the baseline and ROUGE-reward models in all metrics (p < 0.001).
  • Human evaluation finds the ROUGESal+Entail model better than See et al. (2017) on both relevance and readability.The pairwise comparison uses 100 samples.
  • On test-only DUC-2002, ROUGESal+Entail significantly beats both the cross-entropy and ROUGE-reward models on all four metrics with a large margin (p < 0.001).The authors interpret this as evidence of better transferable and generalizable saliency and entailment skills.

7 Output Analysis

Output analyses find that ROUGESal improves salient-information matching, while Entail improves logical entailment; reward-based models also differ in abstractiveness.

  • Saliency Analysis: 30.86% salient-word match was achieved by ROUGESal, exceeding See et al. (2017), the baseline, and ROUGE-reward models.The corresponding Cloze-Q&A saliency scores were 64.66%, and ROUGESal scores were 46.56%; all differences were statistically significant at p < 0.001.
  • Entailment Analysis: 28.98% entailment was achieved by the Entail-reward model, significantly exceeding the other three models at p < 0.001.The comparison used average entailment from ground-truth summary sentences to output summary sentences; similar trends held for document-to-summary entailment.
  • Abstractiveness Analysis: All reward-based reinforcement-learning models produced significantly more novel n-grams than the cross-entropy baseline.Novel n-gram percentage was the measure used for abstractiveness.
  • Abstractiveness Analysis: Entail-reward and ROUGE-reward models significantly improved abstractiveness over See et al. (2017), whereas ROUGESal was comparable.Entail-reward maintained statistically equal abstractiveness to ROUGE-reward, while ROUGESal performed somewhat worse.

8 Conclusion

The paper presents reinforcement-learning rewards for saliency and directed logical entailment, combines multiple rewards through alternate mini-batches, and reports state-of-the-art or strong transfer results.

  • 8 Conclusion: The model uses novel reinforcement-learning rewards to improve saliency and directed logical entailment in abstractive summaries.The approach targets these properties alongside the paper’s broader summarization objectives.
  • 8 Conclusion: Multiple rewards are optimized simultaneously in alternate mini-batches through a novel multi-reward approach.The conclusion identifies this optimization strategy as effective.
  • 8 Conclusion: The methods achieve new state-of-the-art results on CNN/Daily Mail and strong test-only improvements in a DUC-2002 transfer setup.The reported CNN/Daily Mail result includes the paper’s overall state-of-the-art claim.

A.1 Saliency Rewards

The saliency reward modifies summary-level ROUGE-L by weighting tokens according to a learned saliency predictor rather than treating all tokens equally.

  • ROUGE-L formulation: Summary-level ROUGE-L compares a reference summary with a generated summary using precision, recall, and F-score.The reference contains u sentences and m tokens, while the generated summary contains v sentences and n tokens.
  • ROUGE-L formulation: The ROUGE-L formulation uses the union longest common subsequence between each reference sentence and every generated sentence.This construction forms the basis for summary-level matching.
  • Saliency weighting: ROUGESal replaces equal token weighting with saliency-predictor weights when modifying ROUGE-L scores.The predictor assigns a weight η(w) to each token; repeated input-token probabilities are averaged.

A.2 Experimental Setup

The experiments use CNN/Daily Mail for training and validation, DUC-2002 for test-only domain transfer, and SNLI, MultiNLI, and SQuAD to build auxiliary classifiers. Training and evaluation also specify model settings, reinforcement-learning sampling, beam search, and classifier performance checks.

  • Datasets: CNN/Daily Mail contains 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs.
  • Datasets: DUC-2002 is used as a test-only transfer setup with 567 documents and one or two human reference summaries.Pretrained CNN/Daily Mail models are directly tested on DUC-2002 to assess domain transfer capabilities.
  • Auxiliary datasets: SNLI and MultiNLI train the entailment classifier, while SQuAD supplies sentence–salient-phrase pairs for the saliency predictor.
  • Training details: The LSTM-RNNs use hidden size 256, a 50k vocabulary, 128-dimensional embeddings, and shared encoder–decoder embeddings.The encoder and decoder are unrolled for 400 and 100 time steps, respectively; gradients are clipped at 2.0 and Adam uses learning rate 1 × 10^-3.
  • Training details: Reinforcement learning uses only 5000 training samples, fewer than 2% of the data, and inference uses beam search of size 4.The reward-specific γ values are 0.9985 for ROUGE, 0.9999 for Entail and ROUGE+Entail, and 0.9995 for ROUGESal and ROUGESal+Entail.
  • Classifier evaluation: Table 5 evaluates the saliency predictor on SQuAD-based answer-span classification accuracy and the entailment classifier on MultiNLI development-set accuracy.The authors report that the entailment classifier is comparable to state-of-the-art models.
Loading 1804.06451v2…