Source-linked AI summary
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
Yang Gao, Wei Zhao, Steffen Eger
TL;DR
Multi-document summary evaluation is expensive, and existing unsupervised metrics have low correlation with human relevance ratings. SUPERT evaluates summaries against pseudo references of salient source sentences using contextualized embeddings and soft token alignment. It improves correlation over state-of-the-art unsupervised metrics and also serves as a reward for reinforcement-learning summarizers.
Problem
Multi-document summary evaluation is expensive, and existing unsupervised methods have low correlation with human relevance ratings at the summary level.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
SUPERT scores can be used to train RL summarizers with reward pairs created from documents, potentially relieving the data-hungriness problem of existing RL-based summarization systems.
Source: §7 · p. 5
Abstract
from arXiv · showhide
We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18-39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/acl20-ref-free-eval.
1 Introduction
Multi-document summary evaluation is costly, while existing unsupervised metrics struggle to distinguish summaries by relevance. SUPERT uses salient source sentences as pseudo references and improves correlation with human ratings; it is also used to guide reinforcement-learning summarizers.
1 Introduction
- Evaluating machine-generated summaries is laborious and expensive, and multi-document evaluation has required substantial human effort.The cited DUC evaluation required 3,000 hours of human effort.Source: §1 · p. 1
- Existing unsupervised methods have low summary-level correlation with human relevance ratings and struggle to distinguish high- from low-relevance summaries.Source: §1 · p. 1
- SUPERT measures semantic overlap between a summary and a pseudo reference built from salient information in the input documents, using contextualized encoders.The paper describes relevance as coverage of salient source information.Source: §1 · pp. 1–2
- The best SUPERT correlates with human ratings 18–39% better than state-of-the-art unsupervised metrics.The comparison is reported in Kendall’s τ.Source: §1 · p. 2
- SUPERT is used as a reward for reinforcement-learning extractive summarizers, which outperform state-of-the-art unsupervised summarization methods on multiple ROUGE metrics.Source: §1 · p. 2
2 Related Work
Related evaluation work either depends on human-written references, human annotations, or human ratings, while fully unsupervised approaches use source-summary heuristics. Prior methods have limitations for multi-document summary evaluation.
Reference-based Evaluation
- Reference-based metrics such as ROUGE, BLEU, and METEOR compare summaries with human-written references.Later variants add lexical resources, embeddings, or contextualized representations.Source: §2 · p. 2
Annotation-based Evaluation
- Annotation-based evaluation uses human ratings or Summary Content Units, while preference and important-bigram annotations provide other supervision signals.Automated Pyramid construction still relies on human-annotated SCUs for reliable scores.Source: §2 · p. 2
Learned Evaluation
- Learned evaluation functions trained on human scores include a BERT-based metric using 2,500 ratings for 500 CNN/DailyMail summaries, but it correlates poorly with Pyramid scores on multi-document summaries.The BERT-based method was designed for single-document summaries.Source: §2 · p. 2
Unsupervised Evaluation
- Unsupervised approaches use source-summary word-distribution divergence, MMR-inspired heuristics, question answering, or cosine similarity between source and summary embeddings.Earlier heuristics have low summary-level correlation, and the question-generation method leaves its generation procedure and multi-document applicability unclear.Source: §2 · p. 2
3 Datasets, Baselines and Upper Bounds
The experiments use TAC’08 and TAC’09 multi-document datasets, with Pyramid scores as human ground truth. The baseline comparison finds that unsupervised methods lag reference-based upper bounds, and simple embedding-based metrics underperform lexical baselines.
Datasets
- The evaluation uses TAC’08’s 48 topics and TAC’09’s 44 topics, each with ten news articles and machine-generated summaries of at most 100 words.Each topic has four reference summaries, and each generated summary receives a Pyramid score used as human ground truth.Source: §3 · pp. 2–3
Baselines and Upper Bounds
- All baseline methods fall far behind the reference-based upper bounds in average summary-level correlation with Pyramid scores.The comparison reports Pearson’s r, Spearman’s ρ, and Kendall’s τ; the supplied passages do not give table values.Source: §3 · p. 3, Table 1 · p. 3
- Among baselines, B¨ohm19 and CELMo perform worse than lexical methods, suggesting single-document metrics and source-summary embedding cosine similarity are ineffective for multi-document summaries.Baselines include TF-IDF, JS divergence, REAPER, B¨ohm19, and CELMo; ROUGE-1/2 and MoverScore serve as reference-based upper bounds.Source: §3 · p. 3
4 Measuring Similarity with Contextualized Embeddings
The section compares contextualized sentence embeddings and soft alignment for measuring summary relevance against source documents. Soft alignment improves performance over cosine-based embedding comparisons, with MSBERT surpassing lexical baselines.
Measuring Similarity with Contextualized Embeddings
- BERT, RoBERTa, ALBERT, and SBERT sentence embeddings are average-pooled into document and summary vectors, then compared by cosine similarity.CSBERT beats other cosine-embedding metrics but still falls short of lexical-based metrics.Source: §4 · p. 3
- The method adopts word mover’s distance to soft-align tokens across summaries and source documents, allowing one token to align with multiple relevant tokens.The cited motivation is that minimizing token-embedding WMDs can outperform document-embedding cosine similarity for semantic similarity.Source: §4 · p. 3
- WMD-based soft alignment substantially outperforms its cosine-embedding counterparts, and MSBERT outperforms all lexical-based baselines.The authors conclude that soft word alignment on contextualized embeddings performs well for multi-document summary evaluation.Source: §4 · p. 4
5 Building Pseudo References
The section compares strategies for building pseudo-references from source documents, finding that news position bias makes simple early-sentence selection especially effective. Graph-based methods offer alternatives, but do not surpass the simple top-sentence heuristic.
5 Building Pseudo References
- Because WMD-based metrics perform best but show a gap between reference-based and reference-free settings, the section explores source-derived pseudo-references.The comparison motivates seeking pseudo-references to narrow that gap.Source: §5 · p. 4
5.1 Simple heuristics
- Selecting the top 10–15 sentences as pseudo-references outperforms lexical baselines by over 16% and MSBERT by over 4%.The authors interpret this as evidence of position bias in news articles.Source: §5.1 · p. 4
5.2 Graph-based heuristics
- The graph-based alternatives include SBERT LexRank, SBERT clustering, and SBERT-based PacSum, using position-agnostic or position-aware graphs.SBERT clustering uses affinity propagation and selects each cluster center; SPS extends PacSum with SBERT sentence similarity.Source: §5.2 · p. 4
- Top+Clique combines each document’s top-N sentences with central sentences from cliques of remaining sentences that are not highly similar to top-N selections.The method selects a clique’s most semantically central sentence, then retains it only if it is not highly similar to a top-N sentence.Source: §5.2 · pp. 4–5
- Position-agnostic graph methods trail position-aware graphs and full-source pseudo-references, while position-aware graph extraction still underperforms simply selecting top sentences.Table 4 presents the graph-based methods' performance.Source: Table 4 · p. 5, §5.2 · p. 5
6 Guiding Reinforcement Learning
The authors test SUPERT as a reward for the NTD reinforcement-learning summarizer against REAPER and JS rewards, and compare it with YLS15. SUPERT is the strongest tested reward, with performance on par with YLS15 on TAC’08 and significantly better on TAC’09.
6 Guiding Reinforcement Learning
- The experiment compares REAPER, JS, and SUPERT as unsupervised rewards for NTD, alongside the non-RL unsupervised summarizer YLS15.The tested SUPERT version uses SBERT similarity to pseudo-references made from the top 10 TAC’08 or 15 TAC’09 sentences per source document.Source: §6 · p. 5
- SUPERT is the strongest considered reward, helping NTD perform on par with YSL15 on TAC’08 and significantly better on TAC’09; Table 5 averages NTD performance over ten runs.Asterisks indicate significant advantage over non-asterisk results (p < 0.01, double-tailed t-tests).Source: §6 · p. 5, Table 5 · p. 5
- Generated summaries are evaluated with ROUGE, while human evaluation is left for future work.The reported comparison therefore uses ROUGE rather than human ratings of these generated summaries.Source: §6 · p. 5
7 Conclusion
The conclusion presents SUPERT as an unsupervised multi-document summary metric using contextualized embeddings and soft token alignment against salient source sentences. It also reports improved ROUGE-2 when SUPERT rewards train a neural RL summarizer.
7 Conclusion
- SUPERT measures semantic overlap between summaries and top source sentences using contextualized embeddings and soft token alignment, without reference summaries or human annotations.The authors characterize this approach as a simple yet effective way to rate summary quality.Source: §7 · p. 5
- SUPERT-rewarded neural RL summarization yields up to 17% quality improvement in ROUGE-2 over state-of-the-art unsupervised summarizers.The authors describe this result as evidence of SUPERT’s effectiveness in a downstream task.Source: §7 · p. 5
- The authors propose that scores on summaries from infinitely many documents can supply reward pairs for RL training, addressing its data-hungriness problem.This is presented as a potential training approach, not as a separately reported experiment.Source: §7 · p. 5