Source-linked AI summary
Better Rewards Yield Better Summaries: Learning to Summarise Without References
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, Iryna Gurevych
TL;DR
ROUGE-based RL summarizers can receive high metric scores even when their summaries receive low human judgments. This paper learns a reward from human ratings that needs no reference summary at use time. The learned reward correlates better with human judgments, and RL systems trained with it produce summaries rated higher by humans than state-of-the-art systems.
Problem
ROUGE can correlate with human judgments at system level but poorly distinguish good from mediocre or bad summaries, limiting its reliability as an RL reward.
Source: §1 · p. 1Method
The paper learns a reward from human ratings on summaries, using only the document and generated summary as input.
Source: §1 · pp. 1–2Results
The learned reward correlates significantly better with human judgments than existing metrics, and RL systems trained with it produce summaries with higher human ratings than state-of-the-art systems.
Source: §8 · p. 9Takeaways & Limitations
A reward learned from 2,500 human ratings can guide both extractive and abstractive RL summarizers without reference summaries.
Source: §1 · p. 2The learned reward is not necessarily a general evaluation metric: it may target one task rather than rank different summary quality levels or cover multiple summary types and genres.
Source: §2 · p. 3
Abstract
from arXiv · showhide
Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as input. Hence, once trained, it can be used to train RL-based summarisation systems without using any reference summaries. We show that our learned rewards have significantly higher correlation with human ratings than previous approaches. Human evaluation experiments show that, compared to the state-of-the-art supervised-learning systems and ROUGE-as-rewards RL summarisation systems, the RL systems using our learned rewards during training generate summarieswith higher human ratings. The learned reward function and our source code are available at https://github.com/yg211/summary-reward-no-reference.
1 Introduction
ROUGE rewards can mislead RL summarizers because they poorly distinguish good from mediocre or bad summaries. The paper learns a reference-free reward from human ratings and reports improved correlation with human judgments and higher-rated summaries from RL systems.
1 Introduction
- ROUGE performs poorly at summary level, making it difficult to distinguish good summaries from mediocre or bad ones and potentially misleading RL optimization.ROUGE can correlate well with human judgments when scores are aggregated across systems, but the paper notes that many large-scale datasets provide only one reference per document.Source: §1 · p. 1
- The learned reward uses a document and generated summary as input, so trained RL systems can use it without reference summaries.It is learned from human ratings on 2,500 summaries for 500 CNN/DailyMail news articles.Source: §1 · pp. 1–2
- The learned reward correlates significantly better with human judgments than ROUGE and guides extractive and abstractive RL systems to summaries with significantly higher human ratings than state-of-the-art systems.The paper studies summary-level correlation and explores regression and preference-learning formulations alongside different encoders and architectures.Source: §1 · p. 2
2 Related Work
Prior approaches use heuristic or learned rewards, but may require references or lack human-evaluation evidence. This work instead learns an RL reward from human ratings, distinguishing that task from building a general-purpose summary metric.
2 Related Work
- Input-specific RL avoids reference summaries with heuristic rewards, but its performance falls far behind cross-input RL; this work trains both kinds using a reward learned from human ratings.Gao et al. (2019) also learn rewards, but from ROUGE scores rather than human ratings.Source: §2 · p. 2
- Heuristic rewards for extractive or abstractive RL have not shown higher human judgments than state-of-the-art systems, and the cited approaches require reference summaries.Arumae and Liu (2019) use question-answering competency rewards; Kryscinski et al. (2018) encourage novel tokens.Source: §2 · p. 2
- Rewards using external data can achieve state-of-the-art ROUGE, but their human appeal is unclear without human evaluation, and they require references.Pasunuru and Bansal (2018) use a keyphrase classifier and entailment classifier trained on external datasets.Source: §2 · pp. 2–3
- Prior learned evaluation metrics with only non-reference features correlate less with human judgments than ROUGE, while reference-based features marginally outperform it.The paper distinguishes its task: learning a reward to identify the best summaries, rather than a metric that ranks summary quality levels or generalizes across types and genres.Source: §2 · p. 3
3 Summary-Level Correlation Study
The study tests whether common metrics identify human-rated good summaries among five summaries per article. All considered metrics have low correlation, and their precision and recall for identifying good summaries are below .50.
3 Summary-Level Correlation Study
- The study compares summary-level human-judgment correlations for ROUGE, BLEU, METEOR, and cosine similarity using InferSent and BERT-Large-Cased embeddings.The embedding similarities compare generated summaries with reference summaries.Source: §3 · p. 3
- Human judgments come from five independent workers rating each summary’s fluency, redundancy, and overall quality on a −1 to 1 scale; the averaged overall score is ground truth.The dataset contains summaries for 500 randomly sampled CNN/DailyMail articles, with five summaries per article.Source: §3 · p. 3
- A summary counts as good when its average human score is at least 0.5 and it ranks in the top two among the five summaries for its article.The study measures Spearman’s ρ, Pearson’s r, and the precision and recall of identifying good summaries.Source: §3 · p. 3
- Below .50 for both G-Pre and G-Rec across all considered metrics: more than half of identified good summaries are not good, and more than half of actual good summaries are missed.The study reports that hillclimbing on these metrics can hardly guide RL agents toward genuinely high-quality summaries.Source: §3 · pp. 3–4, Table 1 · p. 4
4 Problem Formulation
The paper formulates single-document reward learning as approximating human judgments for available document-summary pairs. It explores matching reward scores by regression or matching human preferences between summary pairs.
4 Problem Formulation
- For single-document summarization, the reward function R(x, y; θ) measures summary quality for document x, with θ denoting its parameters.Training data consists of documents paired with summaries and their human-judgment scores; the target is a reward function close to those scores.Source: §4 · p. 4
- Regression learning measures reward closeness to human judgments using mean squared error.The formulation treats human judgments as the ground-truth reward and minimizes squared differences between predicted and target scores.Source: §4 · p. 4
- Preference learning measures agreement on which of two summaries for the same document has the higher reward.Each document provides N(N − 1)/2 summary pairs; cross-entropy minimizes disagreement between the learned reward and human preferences.Source: §4 · p. 4
5 Reward Learning Model
The reward model uses either an MLP over document and summary embeddings or SimRed, which scores informativeness and redundancy. The MLP can use CNN-RNN, PMeans-RNN, or BERT encoders.
5.1 MLP
- The MLP concatenates document and summary embeddings, with CNN-RNN, PMeans-RNN, or BERT available as encoders.CNN-RNN uses CNN sentence encodings followed by an LSTM; PMeans-RNN applies power means to word embeddings before an LSTM; BERT uses the final-layer [CLS] embedding and a sliding window for overlength inputs.Source: §5.1 · pp. 4–5
5.2 Similarity-Redundancy Matrix (SimRed)
- SimRed scores a summary by combining its informativeness relative to the document with its redundancy.It averages cosine similarities between summary and document sentence embeddings for informativeness, and squared cosine similarities between summary sentence pairs for redundancy; a weighted sum yields the reward.Source: §5.2 · p. 5
- PMeans-RNN and BERT use trainable layers on top when used with SimRed.PMeans and BERT are otherwise fixed in this setup.Source: §5.2 · p. 5
6 Reward Quality Evaluation
The reward models are evaluated with five-fold cross-validation on 2,500 human-rated summaries. BERT+MLP+Pref performs best overall, and its scores rise with human ratings where ROUGE-2 does not distinguish the highest-rated summaries from lower-rated ones.
Experimental Setup
- The reward models are evaluated using five-fold cross-validation on 2,500 human summaries, split 64:16:20 into training, validation, and test sets per fold.The setup uses α = 0.85 for SimRed.Source: §6 · pp. 5–6
Reward Quality
- BERT+MLP+Pref significantly outperforms all non-BERT+MLP models and reference-based metrics (p < 0.05).Preference loss generally performs better than regression loss, with the paper attributing this to more precise ranking from correct summary preferences.Source: §6 · p. 6
- BERT+MLP+Pref assigns higher scores to summaries with higher human ratings, while ROUGE-2 scores are similar for the highest-rated summaries and lower-rated ones.The paper reports similar observations for InferSent-Cosine and BERT-Cosine.Source: §6 · p. 6
7 RL-based Summarisation with Learned Rewards
The study evaluates learned rewards in extractive NeuralTD and abstractive ExtAbsRL systems on CNN/DailyMail. In both settings, learned rewards yield higher human ratings, although ROUGE-based training produces higher ROUGE scores in the abstractive comparison.
7.1 RL-based Summarisation Systems
- NeuralTD uses a neural network to estimate draft-summary value and selects remaining document sentences from a distribution based on candidate-summary values.It uses delayed rewards until sentence selection ends, plus a 0.5 reward for each selected lead sentence; ExtAbsRL instead combines a supervised abstractor with an RL extractor.Source: §7.1 · pp. 6–7
- In ExtAbsRL, the learned reward replaces ROUGE-L for stepwise extractor rewards and supplies the final-step reward; it affects the extractor, not the pretrained abstractor.Stepwise reward is the difference between learned scores for summaries with and without the new sentence; generation stops at 1.2 times the reference-summary token count.Source: §7.1 · pp. 6–7
7.2 Extractive Summarisation
- NeuralTD trained with the learned reward receives significantly higher human ratings than Refresh and ExtAbsRL, while ExtAbsRL receives higher ROUGE scores than both.The human-rating differences are significant versus Refresh (p = 0.0088) and ExtAbsRL (p ≪ 0.01); the learned-reward system’s ROUGE scores are lower than most recent systems.Source: §7.2 · pp. 7–8, Table 4 · p. 8
7.3 Abstractive Summarisation
- Learned rewards yield significantly higher human ratings for ExtAbsRL than ROUGE-based rewards, while ROUGE-based rewards produce higher ROUGE scores.The evaluation sampled 20 CNN/DailyMail test documents and used three raters; redundant-sentence summaries fell from six with ROUGE-L rewards to two with learned rewards.Source: Table 5 · p. 8, §7.3 · pp. 8–9
8 Conclusion & Discussion
A reward learned from a relatively small set of human ratings supports RL summarisation whose summaries receive higher human ratings than state-of-the-art systems. The authors propose testing the method on other summarisation and downstream tasks.
8 Conclusion & Discussion
- Human evaluations show that summaries generated using the learned reward outperform state-of-the-art systems in human judgements.The reward is learned from human ratings of summaries’ overall quality and is used to train both extractive and abstractive systems.Source: §8 · p. 9
- 2,500 human ratings on 500 summaries underpin the reward, while state-of-the-art systems require 287k document-reference pairs for training.The authors describe this as two orders of magnitude more document-reference pairs.Source: §8 · p. 9
- Future work includes testing other summarisation tasks, downstream uses of summaries, and reward learning for other natural language generation applications.Examples include multi-document summarisation, answering questions from summaries, translation, sentence simplification, and dialogue generation.Source: §8 · p. 9