PaperlayerView exact revision on arXiv

Source-linked AI summary

Focus Attention: Promoting Faithfulness and Diversity in Summarization

Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, Ryan McDonald

arXiv:2105.11921v1cs.CL

TL;DR

Faithful summaries remain an unsolved problem, while models often produce low-diversity outputs despite multiple good summaries being possible for a source. The paper introduces FAME to bias generation toward source-topical tokens and Focus Sampling for diverse outputs; on XSum, FAME improves faithfulness and ROUGE, and Focus Sampling outperforms top-k or nucleus sampling for diverse, faithful summaries.

  • Problem

    Faithful summarization remains unsolved, and neural models tend to produce low-diversity outputs despite multiple equally good summaries being possible for a source document.

    Source: §1 · pp. 1–2

  • Method

    FAME dynamically biases the decoder toward tokens topically similar to the input, while Focus Sampling selects topically relevant tokens to generate diverse summaries.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    On XSum, FAME improves faithfulness over vanilla ROBERTAS2S and PEGASUS and overall ROUGE, while Focus Sampling yields more diverse, faithful summaries than top-k or nucleus sampling.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that FAME improves faithfulness and ROUGE, while Focus Sampling is a better alternative to top-k or nucleus sampling for diverse, faithful summaries.

    Source: §6 · p. 9

    Whether BERTFaithful generalizes beyond XSum remains uncertain; the authors call for more work to assess it on other datasets.

    Source: §5 · p. 6

Abstract

from arXiv · show

Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents, outperforming their vanilla counterparts on \rouge and multiple faithfulness measures. We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-$k$ or nucleus sampling-based decoding methods.

1 Introduction

The introduction frames faithful and diverse summary generation as unresolved quality challenges, then presents FAME as a source-conditioned way to steer summaries toward supported, topical content. It reports stronger faithfulness for two XSUM summarizers and positions Focus Sampling as a more faithful, topical route to diverse outputs than top-k or nucleus sampling.

Motivation

  • Faithfulness remains an unsolved summary-quality challenge, while models also tend to produce low-diversity outputs despite multiple equally good summaries for a source document (Kryscinski et al., 2020; Maynez et al., 2020; Gabriel et al., 2020; Fan et al., 2018; Kulikov et al., 2019; Freitag et al., 2020; Choi et al., 2020).The introduction notes that sampling can increase diversity at the cost of quality (Fan et al., 2018; Holtzman et al., 2020).Source: §1 · pp. 1–2

FAME

  • FAME augments contextual representations with a dynamic source-conditioned vocabulary bias so the decoder proactively generates tokens similar or topical to the input.The method is inspired by how humans plan summaries with document-level information in mind.Source: §1 · p. 2

FAME promotes summaries faithful to the source

  • On XSUM, ROBERTAS2S and PEGASUS augmented with FAME produce more faithful summaries than their vanilla counterparts, assessed with multiple faithfulness measures.The authors also introduce BERTFaithful, a scorer trained on manually annotated document-summary pairs as an efficient proxy for human evaluations.Source: §1 · p. 2

FAME enables diverse summaries

  • Focus Sampling steers generation toward topically relevant tokens and yields diverse, topically consistent, faithful outputs more effectively than top-k or nucleus sampling (Fan et al., 2018; Holtzman et al., 2020).Figure 1 contrasts PEGASUS and PEGFAME predictions, and shows diverse samples from top-k, nucleus, and Focus Sampling.Source: Figure 1 · p. 1, §1 · p. 2

Diversity and faithfulness

  • Experiments find that optimizing for high diversity often comes at the cost of faithfulness, and FAME provides a mechanism to trade off faithfulness and diversity.This motivates evaluating both qualities rather than treating diversity alone as the objective.Source: §1 · p. 2

2 Related Work

Prior summarization work adds task-specific priors, document topics, or faithfulness constraints, while diverse-generation methods use varied decoding and training strategies. FAME instead learns document themes through dynamic vocabulary biasing, and Focus Sampling proactively favors input-relevant tokens.

Task-Specific Architectural Priors

Topic-Aware Generation Models

  • Topic-aware summarizers have used LDA or Poisson factor analysis for input topic vectors, whereas FAME dynamically learns a target-induced topic distribution assuming the human summary proxies the input document (Narayan et al., 2018; Ailem et al., 2019; Wang et al., 2020c).Related work also applies document-level topic information to language models and response generation.Source: §2 · pp. 2–3

Faithful Generation Models

  • Faithful-generation approaches condition on extracted facts, jointly generate syntax, learn attention confidence, or use matching and similarity losses, but input-consistent generation remains open (Cao et al., 2017; Song et al., 2020; Tian et al., 2019; Wang et al., 2020d; Gabriel et al., 2020).Source: §2 · p. 3

Diverse Generation Models

  • Diverse-generation work varies beam search, truncates probability tails with top-k or nucleus sampling, or modifies training objectives to favor diverse outputs (Kulikov et al., 2019; Fan et al., 2018; Holtzman et al., 2020; Martins et al., 2020; Welleck et al., 2019a).Source: §2 · p. 3
  • Conditional generation research has emphasized diverse questions and paraphrases, while summarization has used sampled source masks; Focus Sampling instead proactively biases output toward tokens topically relevant to the input (Narayan et al., 2016; Dong et al., 2017; Sultan et al., 2020; Wang et al., 2020b; Li et al., 2016b; Dai et al., 2017; Xu et al., 2018; Cao and Wan, 2020; Gehrmann et al., 2018; Cho et al., 2019).Unlike top-k and nucleus sampling, it applies this bias proactively rather than truncating the tail at every decoding step.Source: §2 · p. 3

3 Summarization with Focus Attention

FAME augments a Transformer encoder-decoder with a source-derived vocabulary bias that encourages topical generation. Summary-induced topic targets and Focus Sampling further steer the decoder toward topical content.

3 Summarization with Focus Attention

  • FAME adds a source-conditioned vocabulary bias to decoder logits, encouraging output tokens similar to the input rather than relying only on the decoder language model.The mechanism provides a short circuit from the source to vocabulary logits.Source: §3 · p. 4
  • Each input token receives a contextual vocabulary distribution, which is averaged across the document to form a source-conditioned vocabulary distribution.The token-level distribution measures contextual similarity between an input token and vocabulary items.Source: §3 · p. 4
  • At each decoding step, encoder-decoder attention weights the token-level distributions to produce a dynamic focus bias for vocabulary predictions.The weighted sum uses attention over source tokens for the output token.Source: §3 · p. 4
  • The topic target uses the human-written summary as a proxy for input topical content, excluding frequent vocabulary tokens to focus on content words.The final objective combines the likelihood and topic losses.Source: §3 · p. 4
  • The target-induced topic distribution is task-agnostic and can favor input tokens or novel tokens depending on the target task.This contrasts with task-specific copying priors, which may not suit highly abstractive extreme summarization.Source: §3 · p. 4
  • Focus Sampling draws k tokens from the topic distribution into a fixed vocabulary subset, with F also included; unlike top-k or nucleus sampling, it can work with beam search.Sampling parameters can control output faithfulness or diversity, and the fixed topical vocabulary is intended to support input-consistent generation.Source: §3 · pp. 4–5

4 Experimental Setup

The experiments assess FAME's faithfulness and Focus Sampling's diversity on summarization and related generation tasks. They compare pretrained model variants and evaluate lexical overlap, semantic similarity, faithfulness, diversity, and output quality.

4 Experimental Setup

  • The setup evaluates whether FAME generates faithful summaries and whether Focus Sampling improves diverse, faithful generation over other sampling-based decoding methods.Source: §4 · p. 5

4.1 Extreme Summarization

  • XSUM's single-sentence, highly abstractive summaries test document-level theme capture; appendix experiments cover CNN/DM and two editing tasks, with FAME achieving SOTA on both editing tasks.Source: §4.1 · p. 5

4.2 Pretrained Models with FAME

  • The study compares ROBERTAS2S and PEGASUS models augmented with FAME, using a newly trained PEGASUS model matched to ROBERTAS2S in parameter count and vocabulary.PEGASUS has pretrained encoder-decoder attention, while ROBERTAS2S has randomly initialized encoder-decoder attention.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • Evaluation includes ROUGE for lexical overlap and BERTScore for candidate-reference semantic similarity.Source: §4.3 · p. 5
  • Faithfulness evaluation uses entailment and QA-based Feqa scores, plus BERTFaithful, a scorer trained on manually annotated document-summary pairs as a surrogate for human assessment.ROUGE and BERTScore are noted as not correlating well with faithfulness; human evaluation is costly and has shortcomings.Source: §4.3 · p. 6
  • Diversity measures include Unique summaries and Distinct-N; output-quality reporting also includes length, repetition errors, and ROUGE-1 precision against the input.Source: §4.3 · p. 6

5 Results

On XSUM, FAME improves fluency, informativeness, and faithfulness over vanilla ROBERTAS2S and PEGASUS, with larger effects for ROBERTAS2S. Focus Sampling yields more faithful and higher-ROUGE summaries than top-k or nucleus sampling, while diversity can trade off against quality.

FAME Summaries Are More Fluent, Informative and Faithful

  • Both FAME models outperform their vanilla counterparts on fluency, informativeness, and faithfulness; PEGFAME has the strongest overall scores, but differences from PEGASUS are not significant on BERTScore and faithfulness measures.FAME is more effective for ROBERTAS2S than PEGASUS on extreme summarization.Source: §5 · p. 7

Generating Diverse and Faithful Summaries with Focus Sampling

  • Focus Sampling has the highest ROUGE scores and generates summaries that entail documents most, while top-k and nucleus sampling have poor entailment; added diversity can reduce faithfulness.Focus Sampling produces 1.61 unique summaries with ROBFAME and 2.77 with PEGFAME; adding top-k or nucleus sampling increases some diversity measures but harms faithfulness again.Source: §5 · p. 7, Table 2 · p. 7
  • Focus Sampling summaries entail documents most and have the highest ROUGE scores, but Divtop,k and Divnucleus produce more unique summaries; Focus Sampling has more diverse unigrams than Divtop,k.For ROBFAME and ROBFAME-Focussample,k, ROUGE-L/entailment scores are 34.81/41.3 and 31.0/34.3, respectively.Source: §5 · pp. 7–8

Focus Attention and Sampling Work Differently in ROBFAME and PEGFAME

  • ROBFAME’s topic distribution is peakier than PEGFAME’s, and it consistently achieves higher ROUGE-1 when generation is restricted to the lower tested top-k vocabularies.Average peakiness scores are 1.25 for ROBFAME and 0.45 for PEGFAME; the comparison covers k values of 50, 100, 200, 500, and 1000.Source: §5 · p. 8, Figure 4 · p. 8, Figure 3 · p. 8

Ablations and SOTA Comparisons

  • Jointly supervised FAME significantly outperforms vanilla baselines, whereas focus attention without topical supervision is not significantly better; the idealized ORACLE results indicate a remaining performance gap.ORACLE assumes the topic distribution matches the reference summary’s token distribution.Source: Table 3 · p. 8, §5 · p. 9
  • PEGFAME outperforms several prior XSUM systems but trails original PEGASUS, which has substantially more parameters.The compared systems include PtGen, ConvS2S, MMN, MASS, and BART.Source: Table 3 · p. 8, §5 · p. 9

6 Conclusion

FAME dynamically biases the decoder toward tokens topically similar to the input. The paper reports improved faithfulness and ROUGE for state-of-the-art abstractive summarizers, and presents Focus Sampling as an alternative for generating diverse, faithful summaries.

6 Conclusion

  • FAME dynamically biases the decoder to proactively generate tokens topically similar to the input.The paper reports that it enhances the faithfulness of existing state-of-the-art abstractive summarization models while improving overall ROUGE scores.Source: §6 · p. 9
  • Focus Sampling is presented as a better alternative to top-k or nucleus sampling for generating diverse, faithful summaries.Source: §6 · p. 9
  • The paper concludes that FAME improves both faithfulness and overall ROUGE for existing state-of-the-art abstractive summarization models.Source: §6 · p. 9

Ethical Considerations

The ethical considerations concern unfaithful or untruthful generation, risks inherited from untrustworthy or biased training data, and malicious misuse. User awareness and measures such as data selection and evaluation are relevant to reducing potential harm.

Ethical Considerations

  • Abstractive targets can require unseen words and are associated with generated content inconsistent with source material.Models may be neither faithful to the source nor factual.Source: Ethical Considerations · p. 9
  • Training on suspect or malicious sources can make a model untrustworthy by teaching it their language and topics.Source: Ethical Considerations · p. 9
  • Training-data biases around gender or race risk being propagated in predictions, especially when data do not represent current norms and practices.Source: Ethical Considerations · p. 9
  • Without user awareness and measures such as better data selection or evaluation, generated text can be damaging.The paper characterizes these considerations as non-malicious model behavior learned from source material.Source: Ethical Considerations · p. 9
  • Models can also be maliciously misused to generate fake news, spam, or other text intended to mislead the public.Source: Ethical Considerations · p. 9

A Implementation and Reproducibility Details

The implementation uses pretrained RoBERTa-based summarizers, with FAME adding randomly initialized focus-attention parameters. The appendix specifies training, vocabulary, decoding, and evaluation-computation settings.

A Implementation and Reproducibility Details

  • ROBFAME has 463M parameters, including 8M additional parameters over ROBERTAS2S, and both use shared encoder-decoder parameters initialized from RoBERTa checkpoints.Focus-attention parameters in ROBFAME are randomly initialized; the PEGASUS-based models use the same configuration except their encoder-decoder attention parameters are pretrained.Source: A Implementation and Reproducibility… · p. 15
  • Models are fine-tuned with Adam at learning rate 0.05, batch size 128, and 200k XSUM or 400k CNN/DM steps, without hyperparameter tuning.The setup uses 40k-step linear warmup and square-root decay; checkpoints are selected by validation ROUGE-L.Source: A Implementation and Reproducibility… · p. 15
  • FAME uses 500 frequent sentence pieces for XSUM and 1,000 for CNN/DM, with beam size 4.Source: A Implementation and Reproducibility… · p. 15
  • Entailment-score computation takes about 20 minutes for both test sets, while Feqa takes 8–12 hours per setting and test set.Entailment scores use Cloud TPU v3 accelerators; Feqa question generation and answering use an NVIDIA V100 GPU.Source: A Implementation and Reproducibility… · p. 15

B Abstractive Summarization Results on CNN/DailyMail

On CNN/DM, PEGFAME outperforms two copy-mechanism models but trails T5, PEGASUS, and ProphetNet; ROBFAME has a small ROUGE-2 regression and lower entailment and Feqa scores than its baseline. The appendix describes CNN/DM as highly extractive, which the authors propose may explain some ROBFAME drops.

B Abstractive Summarization Results on CNN/DailyMail

  • PEGFAME outperforms PtGen and SAGCopy but performs worse than T5, PEGASUS, and ProphetNet on CNN/DM.ROBFAME's ROUGE-2 score is 0.23 points worse than the ROBERTAS2S baseline.Source: Table 4 · p. 15, B Abstractive Summarization Results on… · p. 15
  • CNN/DM summaries are bullet-point highlights with high extraction, while XSUM uses single-sentence, highly abstractive summaries.CNN/DM has 287,227/13,368/11,490 train/validation/test pairs; inputs are truncated to 512 tokens and summaries limited to 128 tokens.Source: B Abstractive Summarization Results on… · p. 15
  • ROBFAME scores worse than ROBERTAS2S on CNN/DM entailment and Feqa; the authors hypothesize that limited copying may contribute.The encoder-decoder attention is not pretrained; Feqa scores may not be fully comparable because ROBFAME summaries are three words longer on average and generate 1.2 more questions.Source: B Abstractive Summarization Results on… · pp. 15–16
  • Table 5 presents faithfulness and qualitative assessments for CNN/DM summaries.Source: Table 5 · p. 16

C Text Editing Results

The section evaluates FAME on two low-abstraction text-editing tasks and controlled top-k summary generation. Editing results vary by dataset, while ROBFAME outperforms PEGFAME at the tested lower k values.

C Text Editing Results

  • The editing experiments use balanced Wikipedia from DiscoFuse for sentence fusion and WikiSplit for split-and-rephrase; both tasks require little abstraction.Models are trained for 300k steps with global batch size 256, and inputs and outputs are padded to 128 tokens, covering all data.Source: C Text Editing Results · p. 16
  • Editing performance is assessed with corpus-level BLEU, exact-match accuracy, and SARI, which evaluates n-gram additions, deletions, and retention against references and the input.The results are presented in Table 6.Source: C Text Editing Results · p. 16, Table 6 · p. 16
  • Vanilla PEGASUS already beats current state-of-the-art on both editing tasks; PEGFAME improves all DiscoFuse metrics, but on WikiSplit it is 0.1 BLEU worse while improving exact match and maintaining SARI.The WikiSplit BLEU comparison is against PEGASUS.Source: C Text Editing Results · p. 16, Table 6 · p. 16

D Controlled Generation with focus attention using Top-k tokens

  • At k values 50, 100, 200, 500, and 1000, ROBFAME consistently outperforms PEGFAME in controlled top-k generation.The passage attributes this to ROBFAME's peaky and PEGFAME's smooth topic distributions; Table 7 also reports ROUGE-2, ROUGE-L, entailment, Feqa, and BERTScores.Source: D Controlled Generation with focus… · p. 16

E Diverse Summarization with Divtop,k, Divnucleus and Focussample,k

The section presents diverse summaries generated with Divtop,k, Divnucleus, and Focussample,k. Its controlled top-k examples also note that smaller k values tend to bring more hallucinations for both model families.

E Diverse Summarization with Divtop,k, Divnucleus and Focussample,k

  • Figure 5 juxtaposes the BBC article and human summary with ROBERTAS2S and PEGASUS predictions, with and without FAME; orange text is unsupported by the article.Source: Figure 5 · p. 17
  • With smaller k, both ROBERTAS2S- and PEGASUS-based models using Focustop,k tend to hallucinate more often.Figure 6 marks text unsupported by the input article in orange.Source: Figure 6 · p. 18
  • Figure 7 shows FAME predictions using Focussample,k with k = 10000, marking unsupported text in orange.Source: Figure 7 · p. 18
  • Figures 8–9 show ROBERTAS2S and ROBFAME outputs with Divtop,k and Divnucleus; Figures 10–11 show the corresponding PEGASUS and PEGFAME outputs.These figures present diverse summaries for the article shown in Figure 5.Source: E Diverse Summarization with Divtop,k,… · p. 16, Figure 8 · p. 19, Figure 9 · p. 20

Related papers

Loading 2105.11921v1…