Source-linked AI summary
Fill in the BLANC: Human-free quality estimation of document summaries
Oleg Vasilyev, Vedant Dharnidharka, John Bohannon
TL;DR
Summary evaluation needs an objective, reproducible alternative to reference-based overlap and costly, subjective human judgments. BLANC measures whether a pretrained language model performs better at understanding a document with its summary, and the paper compares its correlations with human scores and ROUGE while reporting a low correlation for highly compressed summaries.
Problem
ROUGE relies on reference-summary overlap, while human judgments are subjective and less reproducible; score-trained models may also generalize poorly beyond their training domains and styles.
Source: §1 · p. 1Method
BLANC measures how much a pretrained language model’s document-understanding performance improves when it uses the summary, including comparisons between summary-assisted and unassisted reconstructions.
Source: §2.1 · p. 2, §2.3 · p. 3Results
For highly compressed summaries, the reported JS correlation is 0.12, with p-value 0.15; the paper also presents correlations of BLANC-help and ROUGE against annotator scores.
Source: §4 · pp. 6, 8Takeaways & Limitations
BLANC provides objective, reproducible summary-quality measures without requiring human-written reference summaries.
Source: §5 · p. 8On challenging texts and summaries with very low inter-annotator agreement, BLANC’s correlation with human scores is similarly diminished, with borderline p-values.
Source: §4 · p. 7
Abstract
from arXiv · showhide
We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document's text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.
1 Introduction
Summary evaluation balances reproducibility against meaningful assessment: ROUGE uses reference-text overlap, while human judgments are subjective, slow, and expensive. The introduction motivates a reproducible estimator that evaluates how helpful a summary is for understanding its document.
1 Introduction
- ROUGE measures token overlap against human-written references and largely disregards the document’s semantics.The cited alternatives use named entities, other preprocessed features, or embeddings to address this deficiency.Source: §1 · p. 1
- Human evaluation is more meaningful than ROUGE but is subjective, less reproducible, and costly to scale.Evaluators may favor text copied from the document, and more reliable evaluation can require additional prompting or multiple scoring dimensions.Source: §1 · p. 1
- Models trained on human quality scores may inherit the subjectivity of those labels and fail to generalize beyond training domains and styles.The passage says broad generalization may require a massive, representative sample of documents.Source: §1 · p. 1
- Question-answer approaches estimate summary helpfulness, but choosing questions requires document knowledge and limits reproducibility.The possible questions span a vast set, and selecting among them presupposes knowledge of the document.Source: §1 · pp. 1–2
- The introduction proposes a conceptually simple, reproducible approach to summary-quality estimation.The next section is presented as the description of that approach.Source: §1 · p. 2
2 Methods
BLANC estimates summary quality by measuring how much a summary helps a pretrained language model understand a document. It offers two variants: one supplies the summary during inference, while the other fine-tunes the model on it first.
2.1 Introducing BLANC
- BLANC estimates summary quality by measuring how much an independent, pretrained BERT model improves at reconstructing masked document tokens with a summary.The design aims to estimate reader-help across document domains without special text preconditions, using a well-documented, widely used base model.Source: §2.1 · p. 2
- BLANC-help prepends the summary during inference, whereas BLANC-tune fine-tunes the model on the summary before processing the document.The two versions are named BLANC-help and BLANC-tune; tune separates summary learning from document understanding and requires no concatenation during inference.Source: §2.1 · p. 2, §2.3 · p. 4
2.2 BLANC-help
- BLANC-help measures the difference between accuracy unmasking masked sentence tokens with a summary and accuracy unmasking them with a filler concatenated before the sentence; the filler controls input length.Masking aims to evenly cover all sentence tokens at a certain frequency. Lmin = 4 masks only words of 4 or more characters, and M = 6 makes about one-sixth of tokens eligible, reflecting BERT’s 15% masking rate.Source: §2.2 · pp. 2–3, Figure 1 · p. 2
2.3 BLANC-tune
- BLANC-tune fine-tunes on summary-derived masked-token examples, then compares the tuned and original models on identical masked document sentences.The tuning set is extremely small: each sample is the same summary with different tokens masked; preliminary experiments found BLANC-tune and BLANC-help returned similar values.Source: §2.3 · pp. 3–4, Figure 3 · p. 3
2.4 Extractive summaries: no-copy-pair guard
- For extractive summaries, exact copied sentence pairs may artificially help unmasking; a guard skips them, but the paper generally leaves this guard off.The guard is used only for the corner case of copying random sentences from the text.Source: §2.4 · p. 4
3 Basic validation of BLANC measurement
BLANC-help was tested against randomized summaries and under partial sentence replacement. Real generated summaries almost always scored above random-sentence summaries, and scores deteriorated when summary sentences were replaced with random words.
3 Basic validation of BLANC measurement
- The validation used daily-news summaries generated by abstractive UniML, a semi-abstractive model, and extractive LexRank, with human-rated quality varying widely.Random-word summaries were formed from words in the source text and matched the original summary length.Source: §3 · p. 4
- The random-sentence baseline uses document sentences with the no-copy-pair guard’s second version and compensates for summary length by adding sentences when needed.The experiment replaces a copied summary sentence with another sentence rather than simply removing it.Source: §3 · pp. 4–5, Figure 5 · p. 5
- Generated-summary BLANC-help was almost always higher than random-sentence-summary BLANC-help in Figure 5’s right-hand comparison.The random-sentence summaries use guard version 2 and are length-compensated.Source: Figure 5 · p. 5, §3 · p. 5
- BLANC-help deteriorated when one, two, or all three sentences in a three-sentence summary were replaced with same-length random-word text.Figure 6 sorts the summaries by the original summary’s measure.Source: §3 · p. 5, Figure 6 · p. 5
4 Comparison with human evaluation scores
BLANC-help and ROUGE correlate with human ratings comparably, while BLANC requires neither human-written reference summaries nor human quality annotations. The comparisons also show that summary length and annotator agreement affect how these correlations should be interpreted.
4 Comparison with human evaluation scores
- BLANC requires neither human-written summaries nor human-annotated quality scores.This contrasts with reference-based ROUGE, though human-score correlation is only an independent confirmation that the measure is sensible, not full validation.Source: §4 · p. 5
- BLANC-help and ROUGE correlate with annotators as well as or better than about 30% of annotators; shown correlations have p-values of order 10^-6 or lower.Figure 7 compares BLANC-help and ROUGE-L variants with annotator agreement on CNN / Daily Mail summaries.Source: Figure 7 · p. 6, §4 · p. 6
- When inter-annotator agreement is very low, BLANC’s correlation with human scores is similarly diminished and may have borderline p-values.The authors caution that correlation with fallible human scores is an independent confirmation, not full validation.Source: §4 · pp. 5, 7
- Like human ratings, BLANC scores benefit from longer summaries, but BLANC is more sensitive to compression; normalize by compression C when summary length should be excluded.The authors note that longer summaries add proportionally more help, while longer texts add proportionally more masked tokens.Source: §4 · p. 7
- For summaries with compression below 0.05, the Jensen-Shannon measure’s human-score correlation is 0.12, with p-value 0.15.The authors describe that p-value as too high to establish a reliable correlation.Source: §4 · p. 8
- BLANC can be competitive with three annotators across summary qualities, especially for overall quality and fluency.Figure 9 compares BLANC with a group of three annotators against the remaining seven.Source: §4 · p. 8, Figure 9 · p. 8
5 Conclusion
BLANC estimates summary quality by how helpful a summary is for understanding its document, without requiring human-written reference summaries. Unlike ROUGE, it inspects the document itself; the authors plan to explore variations and provide a public package.
5 Conclusion
- BLANC is an objective, reproducible family of summary-quality measures based on how helpful a summary is for understanding its document.It does not require human-written reference summaries.Source: §5 · p. 8
- Unlike BLANC, ROUGE estimates summary quality without inspecting the original document text.The authors question ROUGE’s use for headline generation, where a single best headline may be difficult to identify.Source: §5 · p. 8
- ROUGE’s apparent lower processing burden must account for the human effort required to write its reference summaries.The authors plan to consider BLANC variations and provide a public package called blanc.Source: §5 · pp. 8–9