Source-linked AI summary
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano
TL;DR
The paper addresses limitations of ROUGE, especially its lexical-similarity bias and weak accounting for fluency and readability in abstractive summarization. It proposes QA-based metrics that can operate without human references and uses them as reinforcement-learning rewards, finding improvements over ROUGE-based approaches in automated and human evaluations.
Problem
ROUGE is biased toward lexical similarity and poorly accounts for the fluency and readability of generated abstractive summaries.
Method
The paper develops QA-based evaluation metrics without human annotations and uses them as rewards in reinforcement-learning summarization schemes.
Results
The reinforcement-learning model improves over the state of the art in ROUGE and human assessments, while reference-free metrics bring clear performance benefits.
Takeaways & Limitations
QA-based metrics provide a path toward partially self-supervised summarization training and may benefit from advances in automatic question generation.
Takeaways & Limitations
The QA-based metrics remain a useful complement to ROUGE but still require an additional condition described in the paper's unsupervised-setting extension.
Abstract
from arXiv · showhide
Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from suboptimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compares to ROUGE -- with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as a reward.
1 Introduction
Abstractive summarization needs faithful evaluation beyond lexical overlap, because ROUGE poorly captures fluency and readability. The paper proposes QA-based metrics without human annotations and uses them in reinforcement learning, reporting improvements over ROUGE-based approaches.
- Abstractive summarization is harder than extractive summarization, making faithful evaluation metrics crucial for measuring and driving progress.
- ROUGE is widely used but inadequately accounts for fluency and readability while favoring lexical similarity.
- The paper introduces Question Answering-based metrics that do not require human-generated reference summaries.
- The authors quantitatively compare summarization metrics using correlations with human assessments and qualitatively assess generated summaries with human evaluation.
- The proposed metrics are used as rewards in reinforcement-learning schemes, including in-domain and out-of-domain unsupervised settings.
- Fitting human judgments with carefully chosen measures trains reinforcement-learning models that improve over the state of the art in ROUGE and human assessments.
2 Evaluation Metrics
The paper compares established and proposed metrics for abstractive summarization, emphasizing QA-based measures that assess summary quality without human reference summaries. Human-correlation experiments indicate that unsupervised QA metrics capture readability and relevance better than the reported alternatives.
- Existing and proposed metrics: The evaluation compares TextRank, ROUGE, novelty, language-model perplexity, and QA-based metrics for abstractive summarization.
- Question-Answering based Metrics: QA-based metrics score whether a system retrieves expected answers from questions associated with the source or reference text.The proposed unsupervised variants generate questions and answers from the source article rather than a human reference summary.
- Question-Answering based Metrics: QA-based metrics also account for readability because the QA system must locate answers in variable generated text.
- Question-Answering based Metrics: The unsupervised QA metrics QAfscore(unsup) and QAconf(unsup) assess quality and informativeness without requiring human-generated reference summaries.
- Human correlation analysis: In human-correlation analysis, QAconf and QAfscore captured readability and relevance better than the other reported metrics, including ROUGE.The analysis used Spearman rank correlations against human readability and relevance judgments on CNN/Daily Mail summaries.
- Human correlation analysis: A Ridge-regression aggregate metric performed best when combining ROUGE-L with article-level QAconf and QAfscore.The model was fitted and evaluated through repeated random 50/50 splits of 50 human-evaluated samples.
3 Summarization Models
The paper situates abstractive summarization within sequence-to-sequence and reinforcement-learning approaches, then trains models with rewards derived from its metric study. Its reported evidence favors mixed maximum-likelihood and reinforcement-learning training over reinforcement learning alone.
- Summarization models: Abstractive summarization uses sequence-to-sequence models that generate output text rather than only selecting and concatenating source passages.
- Summarization models: Sequence-to-sequence summarizers can repeat text, motivating coverage mechanisms and Intra-Decoder Attention to reduce duplicate information.
- Reinforcement learning: Negative log-likelihood training is limited by exposure bias and its inability to handle the many potentially valid summaries.
- Reinforcement learning: Reinforcement learning trains summarization models through reward signals, including approaches using REINFORCE and QA-based rewards.
- Mixed training objectives: The proposed training scheme combines supervised and reinforcement-learning objectives, using a reward function informed by the metric study.
- Mixed training objectives: ML+RL performed best for readability and relevance, whereas RL alone performed worse than ML in the reported comparison.
4 Experiments
The experiments replace ROUGE rewards with QA-based metrics in reinforcement-learning summarization, including settings that use unlabeled in-domain and out-of-domain text. These metrics improve automated and human-evaluated quality, with benefits depending on metric weighting and data domain.
- Experimental design: The experiments use mixed supervised and reinforcement objectives, comparing ROUGE-L, learned QA-weighted, and equally weighted QA rewards on a pretrained CNN-DM baseline.The setup also tests in-domain and out-of-domain unlabeled data, including TL;DR.
- Unsupervised settings: Unsupervised QA metrics can use article-level rewards without reference summaries, and in-domain data closer to test documents yields progressively better automated and human-evaluation results.Out-of-domain TL;DR data brings a slight QAlearned improvement, while same-domain and test-set data bring larger gains.
- Reward comparison: QA-based rewards improve all evaluated components—ROUGE-L, QAconf, and QAfscore—whereas ROUGE-L reinforcement improves ROUGE at the cost of lower QA-based metrics.Human inspection also finds ROUGE-L-reinforced summaries low in readability and relevance.
- Human evaluation: Human evaluation finds QA-based reinforcement comparable to state-of-the-art ROUGE while significantly improving relevance, especially with in-domain unsupervised data.The evaluation used 50 CNN-DM test articles, three English speakers, and seven systems.
- Reward weighting: QAlearned and QAequally obtain nearly identical aggregate scores—136.4 and 136.43—but QAequally produces significantly lower readability.The authors attribute this difference to QAequally's stronger ROUGE-L weighting.
5 Conclusions
The paper presents QA-based metrics as reference-free rewards for reinforcement-learning summarization. Experiments indicate benefits from both in-domain and out-of-domain unlabeled data, supporting partially self-supervised training and motivating further work on automated question generation.
- Contributions: QA-based metrics provide promising reinforcement-learning rewards without requiring human reference summaries because they can be computed from the source text.The paper presents this reference-free property as central to its approach.
- Conclusions: The experiments find QA-based rewards beneficial when leveraging both in-domain and out-of-domain unlabeled data.This supports using raw text beyond the supervised training corpus.
- Future direction: The results indicate a path toward partially self-supervised summarization training and suggest benefits from progress in automated question generation.This is presented as a direction implied by the reported results.
- Architectural comparison: Under the same reinforcement setup and data, the See–Gehrmann architecture achieves better ROUGE results than the architecture reported by Paulus et al.The paper presents this as evidence of a possible architectural advantage.