PaperlayerView exact revision on arXiv

Source-linked AI summary

Question Answering as an Automatic Evaluation Metric for News Article Summarization

Matan Eyal, Tal Baumel, Michael Elhadad

arXiv:1906.00318v1cs.CLcs.IR

TL;DR

News summarization commonly relies on ROUGE, but single-reference datasets motivate an alternative evaluation approach. The paper introduces APES, which uses question answering over salient source information, and an abstractive model that optimizes APES while improving ROUGE. APES is intended mainly for news-like documents and requires suitable questions and a reliable QA system.

  • Problem

    Single-reference news datasets limit summary-level correlation between ROUGE and manual evaluation, motivating a more suitable automatic metric.

    Source: §1 · p. 1

  • Method

    APES evaluates summaries by running a trained QA system on them with questions about salient source information, while the proposed abstractive model attends to salient entities.

    Source: §1 · p. 2, §5.2 · pp. 6–7

  • Results

    The APES-optimized model achieves APES 46.1 versus 39.8 and improves all ROUGE metrics by about 1 F-point over the baselines.

    Source: §6 · p. 8

  • Takeaways & Limitations

    APES complements ROUGE for news-article summarization, and the proposed model provides competitive ROUGE scores while optimizing APES.

    Source: §7 · p. 9

    APES is useful mainly for news-like documents of about 1k words focused on named entities, while other genres and longer documents require different question-generation methods.

    Source: §1 · p. 2, §7 · p. 9

Abstract

from arXiv · show

Recent work in the field of automatic summarization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrinsic, evaluation metric for this task, Answering Performance for Evaluation of Summaries. APES utilizes recent progress in the field of reading-comprehension to quantify the ability of a summary to answer a set of manually created questions regarding central entities in the source article. We first analyze the strength of this metric by comparing it to known manual evaluation metrics. We then present an end-to-end neural abstractive model that maximizes APES, while increasing ROUGE scores to competitive results.

1 Introduction

The paper introduces APES, an extrinsic metric for news summarization that evaluates whether summaries answer questions about salient source information. It also presents an APES-optimized abstractive model and examines the metric under conditions involving salient questions and reliable QA systems.

1 Introduction

  • Single-reference news datasets motivate a new automatic metric because summary-level correlation between ROUGE and manual evaluation decreases when only one reference is available.The paper targets datasets such as CNN/Daily Mail, Gigaword, and New York Times.Source: §1 · pp. 1–2
  • The metric is effective only when central questions and a reliable QA model are available, with prior CNN and Daily Mail QA accuracies of 72.4 and 75.8.The paper uses these conditions to assess whether APES is feasible and correlated with manual summary evaluation.Source: §1 · p. 2
  • APES evaluates summaries by measuring the percentage of source-document questions answered correctly when a trained QA system receives each candidate summary.The metric is designed for questions associated with central information in the source document.Source: §1 · p. 2
  • APES is proposed for news articles and leaves extension to other genres for future work.The introduction explicitly identifies this extension as unfinished.Source: §1 · p. 2
  • The paper introduces an abstractive model that optimizes APES by increasing attention to salient entities while also improving ROUGE to a competitive level.Its contributions include APES, correlation analysis with Pyramid and Responsiveness, and software for computing APES and ROUGE.Source: §1 · p. 2

2 Related Work

Related work contrasts intrinsic and extrinsic summary evaluation with neural abstractive and extractive summarization methods. APES automates a question-answering-based evaluation approach while addressing the expense and noise of manual or automated Pyramid-style alternatives.

2.1 Evaluation Methods

  • ROUGE measures candidate-reference overlap through N-grams, longest common subsequence, or skip-bigrams, whereas extrinsic metrics assess performance on related tasks requiring document understanding.ROUGE is an intrinsic metric; question answering is one example of an extrinsic task.Source: §2.1 · pp. 2–3
  • Pyramid weights Summary Content Units across multiple human summaries, while Responsiveness combines content selection and linguistic quality as a manual evaluation standard.Both methods require manual evaluation, and Pyramid is most effective with multiple human-made summaries.Source: §2.1 · p. 3
  • Automated Pyramid evaluation is noisy because it must identify and cluster semantically similar content units through error-prone intermediate operations.The compounded noise in these operations degrades the resulting summary evaluation.Source: §2.1 · p. 3
  • APES automates evaluation of whether summaries enable readers to answer questions, extending an extrinsic evaluation task previously discussed in summarization research.The paper assesses the feasibility of this question-answering approximation.Source: §2.1 · pp. 2–3

2.2 Neural Methods for Abstractive and Extractive Summarization

3 APES

APES evaluates news summaries by testing whether they answer questions about central information, using an automatic QA system. For datasets without intrinsic questions, it generates entity-based fill-in-the-blank questions and reuses a CNN-trained QA model.

3 APES

  • APES computes a summarizer’s score from the average number or percentage of central-information questions answered correctly using its generated summaries.The evaluation takes summaries, question-answer pairs, and an automatic QA system as inputs.Source: §3 · pp. 3–4
  • CNN/Daily Mail supplies questions created by removing entities from manually produced highlights, yielding 1,384,887 fill-in-the-blank questions across 312,085 articles.The dataset was originally built for question answering and later repurposed for summarization.Source: §3 · p. 4

3.1 Using APES as an Evaluation Metric for any News Datasets

  • For news datasets without questions, APES identifies named entities in reference summaries with NER and creates fill-in-the-blank questions whose answers are those entities.The procedure covers names, nationalities, organizations, geopolitical entities, and facilities.Source: §3.1 · p. 4
  • The same QA system trained on CNN/Daily Mail is reused for other news datasets, with trained models released to support reproducibility.The approach is applied to AESOP datasets despite their differing source data.Source: §3.1 · p. 4

4 APES on the TAC2011 AESOP Task

On TAC 2011 AESOP, APES is evaluated against manual Pyramid and Responsiveness scores using CNN-trained QA, restricted to the topic segment with entity density comparable to CNN/Daily Mail. APES shows stronger or complementary behavior relative to ROUGE, while single-reference correlations remain noisy.

4 APES on the TAC2011 AESOP Task

  • APES uses CNN-trained QA and automatically generated questions on AESOP because the dataset is too small to train an optimal AESOP-specific QA model.The authors nevertheless report that this incomplete setup produces valuable results.Source: §4 · p. 4
  • Evaluation is restricted to 204 Investigations and Trials documents because their average 3.35 named entities per summary is closer to CNN/Daily Mail than the other TAC topics.CNN/Daily Mail summaries average 6 named entities, while the other TAC topics average 0.65.Source: §4 · p. 4
  • APES shows better correlation with Pyramid and Responsiveness than R-1, R-2, R-L, and R-SU on the evaluated AESOP summaries.Table 1 reports Pearson correlations against the two manual metrics.Source: §4 · p. 5, Table 1 · p. 5
  • APES correlates with ROUGE metrics around 0.6, whereas ROUGE metrics correlate with one another around 0.9, suggesting APES supplies complementary information.The authors note that statistical significance is difficult to establish because the dataset is small.Source: §4 · p. 5, Table 2 · p. 5
  • APES is highest in 7 of 16 assessor-level Pyramid and Responsiveness trials, compared with 6 for R1 and 2 for RL.These comparisons concern evaluations against individual assessors and single references.Source: §4 · p. 5
  • Single-reference correlations are extremely noisy across APES and ROUGE, limiting the reliability of evaluations based on one reference summary.APES is not immune because reducing references also reduces the number of available questions.Source: §4 · p. 5

5 Model

The model targets APES by prioritizing salient entities in summaries, extending a neural abstractive baseline with entity-aware attention and beam-search scoring. It combines entity supervision with standard summarization training and uses predicted entity alignments to favor hypotheses covering central source content.

5 Model

  • Existing systems mention about 2.6 salient entities per summary versus 4.9 in references, despite averaging five named entities versus six.Adding too many entities can reduce QA accuracy by increasing the number of possible answers and distracting the QA system.Source: §5 · pp. 5–6

5.1 Baseline Model

  • The baseline combines negative-log-likelihood training with coverage and length penalties during beam search.Coverage discourages repeated attention and favors broader source coverage, while length normalization compensates for beam search’s preference for shorter outputs.Source: §5.1 · p. 6

5 Model

  • The model extends a neural abstractive baseline to incorporate more salient source entities and maximize APES.The baseline uses an encoder-decoder architecture with attention, while the proposed model adds entity-focused components.Source: §5.1 · p. 6

5.2 Entities Attention Layer

  • The entities attention layer estimates each source token’s independent probability of belonging to a salient entity using sigmoid attention.Unlike softmax alignment, sigmoid scoring does not normalize probabilities across source tokens.Source: §5.2 · pp. 6–7
  • An additional binary-cross-entropy term supervises salient-entity detection alongside the standard log-likelihood loss.The combined objective learns the entity attention layer while retaining the model’s summarization ability.Source: §5.2 · p. 7

5.3 Entities Attention and Beam Search

  • At test time, predicted entity alignments add a beam-search penalty for summaries that fail to attend to source regions judged central.The method passes entity alignments into beam search to favor hypotheses attending salient entities, with Fig. 4 illustrating the resulting attention and word-selection effects.Source: §5.3 · p. 7

6 Results

APES evaluation uses QA performance to assess summaries, with source-document QA establishing a 61.1% realistic upper bound and the APES-optimized model outperforming baselines while improving ROUGE. Error analysis links perfect APES scores to denser salient-entity repetition.

APES evaluation

  • 61.1% APES on full source documents provides a realistic upper bound for the metric.Gold summaries achieve very high APES, while source documents require locating answers among distractors.Source: §6 · pp. 7–8
  • APES detects incoherence in shuffled gold summaries even when salient entities remain present, unlike ROUGE-1.This indicates sensitivity to coherence, readability, and meaning rather than entity presence alone.Source: §6 · p. 8

Abstractive model results

  • 46.1 vs. 39.8 APES, while all ROUGE metrics improve by about 1 F-point over the baselines.The comparison covers See et al. (2017), the authors’ baseline, and the APES-optimized model.Source: §6 · pp. 7–8

Oracle saliency

  • 46.3 versus 46.1 APES with oracle salient-entity positions shows predicted saliency is sufficient to drive summarization.The oracle setting yields only a slight improvement over the model’s predicted positions.Source: §6 · p. 8

Error analysis

  • Perfect-APES summaries have salient-entity density 4.9 versus 3.6 for zero-APES summaries, suggesting repetition helps mark salient entities.The analysis compares 1,630 perfect-APES summaries with 1,691 zero-APES summaries.Source: §6 · pp. 8–9

7 Conclusion

The paper concludes that APES evaluates news summaries through their ability to answer questions about salient information and complements ROUGE. It also presents an abstractive model that optimizes APES by attending to salient entities while retaining competitive ROUGE scores.

7 Conclusion

  • APES evaluates news summaries by testing whether they answer questions about salient information from the source.The metric is presented as useful for news datasets with roughly 1k-word documents focused on named entities.Source: §7 · p. 9
  • The abstractive model optimizes APES by attending to salient entities and achieves competitive ROUGE scores on CNN/Daily Mail.Source: §7 · p. 9

A Experiment Settings

The experiments use a bidirectional LSTM encoder and LSTM decoder, train with Adagrad and staged fine-tuning, and apply length, coverage, and repetition controls during decoding. APES evaluation takes about 40 minutes for 11,490 generated summaries on one process.

A Experiment Settings

  • The model uses a bidirectional LSTM encoder, an LSTM decoder, 128-dimensional embeddings, and a shared 50k-word vocabulary without pretrained embeddings.The encoder has 256-dimensional hidden states per direction, and the decoder has 512-dimensional hidden states.Source: A Experiment Settings · p. 10
  • Training uses Adagrad with learning rate 0.15, gradient clipping at 2, and five fine-tuning epochs after 20 baseline epochs, with λ = 0.01.Training documents are truncated to 400 source tokens and 100 target tokens.Source: A Experiment Settings · p. 10
  • At test time, full source documents are available, beam search uses coverage, length, and repetition penalties, and repeated trigrams are blocked.The model uses α = 0.9, β = 0.5, and γ = 0.5 for decoding penalties.Source: A Experiment Settings · pp. 10–11
  • Evaluating APES on 11,490 generated test summaries takes about 40 minutes using a single process.Source: A Experiment Settings · p. 11

Related papers

Loading 1906.00318v1…