PaperlayerView exact revision on arXiv

Source-linked AI summary

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

Wen Xiao, Iz Beltagy, Giuseppe Carenini, Arman Cohan

arXiv:2110.08499v2cs.CL

TL;DR

PRIMERA addresses the limited fit of existing pretrained models and dataset-specific architectures for multi-document summarization. It combines concatenated-document processing with entity-based masked sentence pretraining, and outperforms prior pretrained and dataset-specific models across six datasets and three domains, including zero-, few-, and full-supervision settings. The paper also notes limits in controllable output length and factuality guarantees.

  • Problem

    Existing approaches use dataset-specific architectures or pretraining objectives that do not target cross-document summarization, increasing reliance on specialized modeling or labeled data.

  • Method

    PRIMERA concatenates related documents in an efficient encoder-decoder transformer and uses Entity Pyramid masking to pretrain cross-document salient-information aggregation.

  • Results

    PRIMERA outperforms prior state-of-the-art pretrained and dataset-specific models on six summarization datasets from three domains across zero-shot, few-shot, and full fine-tuning settings.

  • Takeaways & Limitations

    PRIMERA provides a general pretrained model for multi-document summarization that minimizes dataset-specific modeling and performs strongly across data regimes.

  • Takeaways & Limitations

    In zero-shot inference, PRIMERA controls summary length only through a decoding-time length limit, and like existing generation models has no guarantee of factual content.

Abstract

from arXiv · show

We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins. The code and pre-trained models can be found at \url{https://github.com/allenai/PRIMER}.

1 Introduction

PRIMERA addresses the mismatch between multi-document summarization and existing pretrained models by combining efficient document processing with a pretraining objective tailored to cross-document information aggregation. Across six datasets and three domains, it outperforms prior pretrained and dataset-specific models, especially with limited fine-tuning data.

  • Existing approaches rely on domain-specific information or customized architectures, while existing pretrained objectives are usually single-document or encoder-only.
  • PRIMERA uses Entity Pyramid masking to teach a pretrained encoder-decoder model to identify and aggregate salient information across document clusters.
  • PRIMERA outperforms prior pretrained and dataset-specific models in zero-shot, few-shot, and full fine-tuning settings across six datasets from three domains.
  • With as few as 10 examples, PRIMERA surpasses prior state-of-the-art models by up to 5 ROUGE-1 points.

2 Model

PRIMERA concatenates related documents and processes them with an efficient encoder-decoder transformer, then pretrains with masked sentence generation guided by cross-document entity salience. Entity Pyramid favors representative information shared across documents over exact sentence matches.

  • 2.1 Model Architecture and Input Structure: PRIMERA concatenates documents and uses LED with linear input-length complexity, document-separator tokens, and global attention to share information across documents.LED uses sparse local-plus-global encoder attention and full decoder attention; separator tokens mark document boundaries.
  • 2.2 Pretraining objective: Gap Sentence Generation masks selected sentences and trains the decoder to generate their concatenation as a pseudo-summary.Masked sentences are replaced by a single [sent-mask] token, requiring reconstruction from information in the remaining documents.
  • 2.2 Pretraining objective: Naively extending sentence-overlap scoring to multiple documents can select redundant exact matches rather than representative information.Entity Pyramid instead targets sentences associated with high-frequency entities and representative cluster content.
  • 2.2 Pretraining objective: Entity Pyramid selects salient sentences by favoring information represented across more documents in a cluster.The strategy is inspired by Pyramid Evaluation, where repeated content across references is treated as more salient.

3 Experiment Goals

The experiments evaluate PRIMERA against pretrained and state-of-the-art systems across data regimes, datasets, and model components. They also test whether Entity Pyramid improves salient-content selection and summary quality.

  • The study compares PRIMERA with existing pretrained generation models in zero-shot and few-shot settings and with current state-of-the-art models in fully supervised training.
  • Ablations measure the contributions of PRIMERA’s input structure, pretraining, and masking strategy.
  • Human evaluation assesses whether PRIMERA captures salient information and produces fluent summaries.

4 Experiments

PRIMERA is evaluated across zero-shot, few-shot, and fully supervised multi-document summarization settings using datasets spanning multiple domains. It generally outperforms pretrained and dataset-specific baselines, with especially strong performance when labeled data are scarce.

  • 4.3 Zero-Shot Results: Zero-shot inference controls output length using the average length of gold summaries, while alternative length-control methods are left for future work.The authors consider approximate knowledge of expected summary length reasonable in practice.
  • 4.3 Zero-Shot Results: PRIMERA achieves substantial improvements over three pretrained baselines on most datasets in zero-shot evaluation, with stronger benefits on datasets containing longer summaries.The compared models are BART, PEGASUS, and LED; the authors attribute LED’s weaker performance partly to incompletely pretrained positional embeddings.
  • 4.4 Few Shot Evaluation: PRIMERA outperforms all baselines on every dataset with 10 and 100 examples and converges faster with as few as 10 training examples.Few-shot results average five runs over different subsets, using shared seeds across models.
  • 4.5 Fully Supervised Evaluation: PRIMERA achieves state-of-the-art results on Multi-News, WCEP, and arXiv, while slightly underperforming prior work on Multi-XScience in R-1.The authors suggest that Multi-XScience contains less overlapping information between source documents than PRIMERA’s pretraining corpus.
  • 4.5 Fully Supervised Evaluation: PRIMERA outperforms LED on arXiv while using a sequence length of 4K instead of 16K, indicating effectiveness for documents containing multiple sections.The comparison is made in the fully supervised setting.

5 Ablation Study

Ablations on Multi-News examine the document-input structure and pretraining objective in few-shot settings. Both the input structure and Entity Pyramid masking contribute to performance.

  • Input structure: Few-shot ablations find benefits from both pretraining and the input structure using document-separator tokens and global attention on those tokens.These components are evaluated on Multi-News.
  • Sentence masking strategy: The Entity Pyramid strategy clearly improves few-shot performance over PEGASUS masking when architecture and pretraining data are held constant.The comparison uses the same model architecture and amount of pretraining data, changing only the masking strategy.

6 Human Evaluation

The human evaluation measures content coverage with Pyramid Evaluation and fluency across grammaticality, referential clarity, and structure and coherence. PRIMERA achieves the strongest overall fluency results, while PEGASUS matches it on grammaticality.

  • Settings: Three annotators evaluate both content coverage and fluency using anonymized system summaries and randomized presentation order.Fluency is scored from 1 to 5 for grammaticality, referential clarity, and structure & coherence.
  • Pyramid Evaluation: Pyramid Evaluation measures whether candidate summaries cover weighted Summary Content Units from TAC and DUC references.Three annotators make binary coverage decisions for each SCU, and covered SCU weights are summed into a raw score.
  • Pyramid Evaluation: Because simplified annotations lack the total number of generated-summary SCUs, the evaluation computes length-aware Recall, Precision, and F-1 scores.This adjustment accounts for the lengths of both gold references and system-generated summaries.
  • Fluency Evaluation: PRIMERA achieves the best fluency performance on both datasets across all evaluated aspects.The evaluated aspects are grammaticality, referential clarity, and structure & coherence.
  • Fluency Evaluation: PEGASUS matches PRIMERA’s top performance specifically on grammaticality.

7 Related Work

Prior multi-document summarization methods are mainly graph-based or hierarchical, while pretrained summarization models and efficient transformers address related but incomplete aspects of the task. PRIMERA combines efficient long-input encoding with a task-inspired cross-document generation objective.

  • Neural Multi-Document Summarization: Graph-based models often require auxiliary information, whereas hierarchical models build document-level representations before aggregating across documents.
  • Pretrained Models for Summarization: PEGASUS applies Gap Sentence Generation to abstractive summarization, but its pretraining focus is primarily single-document input.
  • Pretrained Models for Summarization: Longformer and BigBird process long sequences with efficient attention, while CDLM extends Longformer pretraining cross-document but remains unsuitable for generation.
  • Pretrained Models for Summarization: PRIMERA pretrains an efficient encoder-decoder transformer with a task-inspired objective for multi-document summarization.

8 Conclusion and Future Work

PRIMERA is a general pretrained model for multi-document summarization that reduces dataset-specific modeling through entity-based sentence masking. It outperforms prior pretrained and dataset-specific models across six datasets and multiple fine-tuning regimes, with human evaluations also showing top performance.

  • Conclusion: PRIMERA outperforms prior pretrained and dataset-specific models on six summarization datasets from three domains across zero-, few-shot, and full fine-tuning settings.
  • Conclusion: PRIMERA uses a Longformer model with entity-based sentence masking to connect and aggregate information across input documents.
  • Conclusion: Human evaluation also reveals PRIMERA’s top performance.
  • Future Work: In zero-shot inference, output length can be controlled only through a decoding-time length limit.The authors identify controllable generation with desired length supplied as input as future work.

Ethics Concern

The paper’s broader evaluation and implementation context spans multiple datasets, pretrained baselines, and long-input model settings. It also notes practical boundaries involving dataset sampling, input construction, and pretraining resources.

  • Implementation Context: Pretraining uses 100K steps with early stopping and takes approximately seven days on four A100 GPUs.
  • Evaluation Context: The experiments use dataset-specific sampling and truncation choices, including at most 10 documents for WCEP clusters and limited test examples for Wikisum.
  • Evaluation Context: PRIMERA’s evaluation covers datasets spanning scientific, news, Wikipedia, and related-work domains, with several datasets adapted into document clusters.
  • Compared Models: Compared baselines include BART, PEGASUS, and LED, with LED serving as the initial model before PRIMERA’s pretraining.
  • Implementation Context: PRIMERA concatenates documents and uses LED’s efficient encoder-decoder architecture to process long inputs.

C.1 Detailed Experiment for Input Length Limit

The study tunes pretrained baselines’ input lengths in a 10-example Multi-News setting and specifies optimization details for few-shot and fully supervised experiments.

  • Input length selection: BART and PEGASUS are compared at input limits of 512, 1024, and 4096 using five few-shot runs on different shared subsets.The experiment uses 10 Multi-News examples for each input-length configuration.
  • Input length selection: BART performs best with a 1024-token limit, while PEGASUS’s best input length is selected from the same comparison.The supplied passage explicitly identifies BART’s best setting but truncates before stating PEGASUS’s selected limit.
  • Optimization settings: Full supervised experiments use Adam with a 3e-5 scheduled learning rate, batch size 16, and dataset-dependent training and warm-up steps.The detailed step counts are provided in Table 7.

E Detailed Results in Few-shot Setting

This section provides the exact ROUGE scores underlying the few-shot comparison presented in Figure 5.

  • Few-shot results: Table 8 reports detailed R-1/R-2/R-L scores for all datasets in the few-shot setting.The table corresponds to the results visualized in Figure 5.

F Detailed Analysis on Fully Supervised Experiments

The fully supervised analysis reports PRIMERA’s benchmark results, comparisons with prior models, and qualitative examples from Multi-News and DUC2007.

  • Fully supervised benchmark results: PRIMERA achieves state-of-the-art results on Multi-News, WCEP, and arXiv, while slightly underperforming prior work on Multi-XScience R-1.The comparison uses results from prior work on the corresponding datasets.
  • Multi-News analysis: On Multi-News, PRIMERA is around 1 ROUGE point higher than BART-Long, which uses the same model structure.The passage attributes the gap possibly to model structure or pretraining, highlighting document separators, global attention, and multi-document pretraining as design differences.
  • Evaluation scope: The fully supervised evaluation omits Wikisum because of limited computational resources.The same scope limitation is stated in the accompanying footnote.
  • Long-document analysis: PRIMERA outperforms LED on arXiv while using a sequence length four times shorter: 4K versus 16K.This result is presented as evidence that the approach can also summarize single documents with multiple sections.
  • Qualitative examples: Tables 13 and 14 provide generated-summary examples from Multi-News and DUC2007 for PRIMERA and comparison models.The DUC2007 examples use models trained with 10 examples, while Table 13 varies the number of training examples.
Loading 2110.08499v2…