Source-linked AI summary
Efficient Attentions for Long Document Summarization
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, Lu Wang
TL;DR
Long-document summarization is constrained by the quadratic cost of Transformer attention and by efficient methods that do not directly support encoder-decoder attention. The paper introduces HEPOS, a head-wise strided attention, and the GOVREPORT benchmark, reporting longer-input processing, stronger ROUGE results, and more informative, faithful summaries.
Problem
Transformer attention has quadratic computational and memory complexity, while prior efficient attentions do not directly apply to encoder-decoder attention for long-document summarization.
Method
The paper proposes HEPOS, which uses head-wise positional strides to cover source tokens efficiently, and introduces the GOVREPORT dataset for evaluation.
Results
HEPOS-based models process more than 10K input words, achieve significantly higher ROUGE scores on GOVREPORT, new state-of-the-art results on PubMed, and stronger human-rated informativeness and faithfulness.
Takeaways & Limitations
Efficient encoder-decoder attention can support substantially longer inputs while producing more informative summaries with fewer unfaithful errors.
Abstract
from arXiv · showhide
The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.
1 Introduction
Long-document summarization is difficult because Transformer attention scales quadratically, while existing efficient methods do not directly support encoder-decoder attention. The paper proposes HEPOS, introduces GOVREPORT, and reports stronger automatic and human-evaluated summaries.
- Quadratic Transformer time and memory costs make long-document summarization expensive, especially for documents with thousands of words.
- Existing efficient attentions target encoder self-attention but do not directly apply to encoder-decoder attention, which dynamically pinpoints salient source content during decoding.
- HEPOS uses head-wise positional strides to reduce costs while emphasizing important tokens and preserving global context per head.
- GOVREPORT contains about 19.5k government reports with expert-written abstractive summaries, including documents averaging 9.4k words and summaries averaging 553 words.
- More than 10K input words can be processed with HEPOS and sparse encoder attention, versus at most 5K for full encoder-decoder attention models.
- The models achieve significantly higher ROUGE scores on GOVREPORT, new state-of-the-art results on PubMed, and more informative and faithful summaries in human evaluation.
- APESsrc is introduced as a faithfulness metric that correlates better with human judgment than the original APES and an entailment-based scorer.
2 Prior Work on Efficient Encoder Attentions
Prior work reduces Transformer attention costs through fixed patterns, adaptive spans, global tokens, random or hashed connections, learned permutations, and low-rank projections. These approaches trade full pairwise attention for lower memory complexity and, in some cases, additional parameters.
- Full Transformer attention has quadratic memory because each of n document tokens participates in pairwise attention calculations.
- Fixed Patterns: Sliding-window attention restricts each query to neighboring tokens, reducing memory complexity to O(nw).
- Fixed Patterns: Adaptive span learns an independent attention window for each head, with complexity O(nẇ) and O(1) newly learned span parameters.
- Fixed Patterns: Global-token attention adds symmetric full-sequence connections for selected tokens, producing memory complexity O(2ng).
- Fixed Patterns: Stride and random attentions capture longer interactions by attending to every s-th token or r random keys, with complexities O(n^2/s) and O(nr).
- Linformer projects keys and values from n dimensions to k, achieving O(nk) complexity while adding O(n) projection parameters.
- LSH and Sinkhorn attentions use hashing or learned block permutations to combine local context with broader interactions at reduced complexity.
3 Encoder-decoder Attention with Head-wise Positional Strides (Hepos)
HEPOS is an encoder-decoder attention that assigns heads different positional strides and starting positions, allowing them collectively to cover the source while reducing computation and memory.
- HEPOS assigns separate attention heads to different fixed-interval subsets of source tokens, with varying starting positions across heads.
- All HEPOS heads collectively attend to the full input sequence while individual heads follow strided patterns.
- HEPOS is motivated by redundant attention heads and the observation that individual heads rarely attend to several consecutive tokens.
- Each query attends to n/s_h tokens per head, yielding memory complexity O(mn/s_h), where m is output length.
- Linformer is adapted for encoder-decoder attention, whereas pattern-based attentions are not adapted because local grouping makes salient-content pinpointing difficult.
4 GOVREPORT Dataset
GOVREPORT is a large-scale benchmark of long U.S. government reports with expert-written summaries. Its documents are substantially longer than prior datasets, and summary-worthy content is distributed throughout the source.
- GOVREPORT contains 19,466 reports from the GAO and CRS, each paired with a human-written summary.
- The dataset includes 17,519 training samples, 974 validation documents, and 973 test samples after quality inspection and date-based splitting.
- GOVREPORT documents and summaries are significantly longer than those in existing long-document summarization datasets.
- More than 18% of new summary-worthy bigrams occur in the later half of GOVREPORT articles, indicating a more even distribution than in ARXIV and BIGPATENT.
5 Summary Evaluation with Cloze QA
The paper evaluates faithfulness with QA-based and entailment-based metrics, introducing APESsrc to compare answers extracted from summaries with answers extracted from source context.
- Evaluation: APESsrc correlates better with human judgment than APES and an entailment-based scorer.The paper also evaluates FactCC, which predicts an entailment score between the source and summary.
- QA-based Evaluation: APESsrc extends APES by comparing QA answers from the system summary with answers from source context.Questions are cloze questions generated from reference summaries by masking entities, events, dates, and numbers.
- QA-based Evaluation: The metric constructs source context by greedily selecting sentences that maximize ROUGE-2 recall against the reference summary.Examples without the reference answer in the selected context are excluded from QA-model training.
- QA-based Evaluation: APESsrc computes unigram F1 between the system-summary answer and the source-context answer, whereas APES compares the system answer with the reference answer.The source-derived answer is intended to better reflect faithfulness when models rewrite names or numbers incorrectly.
6 Experimental Results
Experiments compare efficient encoder and encoder-decoder attentions under matched-length and matched-GPU settings. Models using HEPOS process more input and generally achieve stronger automatic and human evaluation results.
- 6.2 Comparing Attention Variants: Learnable encoder patterns perform best among encoder variants, with Sinkhorn attention consistently obtaining better ROUGE scores.Combining fixed-pattern techniques is more effective than window-only sparse attention, but increases memory cost.
- 6.2 Comparing Attention Variants: HEPOS consistently yields higher ROUGE scores than Linformer on both datasets with either full or Sinkhorn encoders.With a Sinkhorn encoder, HEPOS matches the variant using full encoder attention.
- 6.3 Reading More Input Boosts Informativeness: Models that read more text obtain higher ROUGE scores on GovReport and PubMed, while adding HEPOS nearly doubles the words processed.HEPOS with an LSH encoder achieves new state-of-the-art results on PubMed and the best models outperform competitive abstractive systems on arXiv.
- 6.4 Reading More Input Improves Faithfulness: Reading more text significantly improves informativeness and reduces fabricated content in human evaluations.HEPOS with a Sinkhorn encoder receives better informativeness scores, and efficient-attention models reduce unfaithfulness, especially hallucination errors.
- 6.4 Reading More Input Improves Faithfulness: The improvements are strongest for later sections such as PubMed Results and Conclusions and GovReport’s “What GAO recommends.”The full-attention model tends to fabricate numbers, whereas the proposed models correct them.
- 6.4 Reading More Input Improves Faithfulness: Longer inputs also produce more faithful summaries under automatic evaluation, although the differences are less pronounced.The comparison uses FactCC, APES, and APESsrc.
- 6.5 Metric Correlation: APESsrc is stronger than APES because source-based answers can capture paraphrases or acronyms that strict lexical matching misses.For example, APESsrc recognizes “mortality” as matching the reference answer “death rate.”
7 Additional Related Work
Prior long-document summarization work spans multiple domains but often lacks datasets suitable for neural training or relies on extractive and truncated-input strategies.
- 7 Additional Related Work: Long-input summarization has been studied for books, patents, movie scripts, and scientific publications, but datasets are often too small for neural models.ARXIV and PUBMED were introduced as large-scale scientific-document datasets.
- 7 Additional Related Work: Extractive methods identify salient sentences using positional information or combined global and local contexts.Their summaries are described as often redundant and incoherent.
- 7 Additional Related Work: Extract-then-abstract systems first select relevant sentences and then rewrite them into abstracts.Other approaches use hierarchical encoders or multiple agents to process document segments, but both cited works truncate articles.
8 Conclusion
The paper proposes HEPOS for efficient long-document encoder-decoder attention and evaluates its effects on coverage, summary quality, and faithfulness.
- 8 Conclusion: HEPOS uses head-wise positional strides to identify salient content while enabling models to process at least twice as many words.Its reported advantages are supported by both automatic and human evaluation.
- 8 Conclusion: HEPOS-based models produce more informative summaries with fewer unfaithful errors.The paper also reports that its cloze QA metric correlates better with human judgment than prior faithfulness metrics.
A GovReport Dataset Collection and Processing
The paper constructs GovReport and related evaluation resources from government and scientific documents, using parsed expert summaries and information-extraction models for structured analysis.
- Dataset collection: GovReport collection retains government reports with expert-written summaries after parsing and length-based filtering.GAO reports are kept when web-page highlights are available, while 10% of the shortest reports are removed to reduce parsing-related errors.
- Dataset collection: CRS reports are collected from EveryCRSReport and restricted to documents with expert-written summaries.
- Evaluation models: Evaluation models use fine-tuned BERT classifiers for entailment labels and answer-span prediction.GovReport uses uncased BERT, whereas PubMed uses a model fine-tuned on PubMed abstracts.
- Information extraction: OneIE extracts entities, relations, and events from reference summaries using unified information extraction models.The government-report model is trained on ACE 2005, while the scientific-domain model uses Genia and PubMed resources.
- Information extraction: Biomedical extraction training combines Genia 2011, Genia 2013, and PubMed, with Genia emphasizing events and PubMed covering more entities.
C Additional Sample Outputs
The paper provides GovReport and PubMed examples showing that HEPOS-based models produce more faithful and informative summaries than truncated-input comparisons.
- Qualitative examples: HEPOS models generate more faithful and informative summaries in qualitative examples from GovReport and PubMed.The examples are presented in Figures 7 and 8.
D Human Evaluation Guideline
Human evaluation measures summary coverage and faithfulness at sentence and aspect levels across government reports and scientific papers. Annotators classify summary aspects, label error types, and rate informativeness against structured references.
- Evaluation setup: Human evaluation uses system summaries, source documents, and corresponding human-written references.
- Aspect labeling: Government-report sentences receive labels for study motivation, findings, or recommendations, while scientific-paper sentences receive four section-level aspect labels.
- Faithfulness errors: Annotators assign sentence-level faithfulness labels for fabricated, contradictory, ambiguous, or otherwise unfaithful content.A zero indicates no error of a given type, and one indicates its presence.
- Faithfulness errors: Hallucination is fabricated content that cannot be found or inferred from the source.
- Faithfulness errors: M misconstruction includes deletions that make sentences incomplete, contextually missing, or ungrammatical.
- Faithfulness errors: False concatenation is treated as misconstruction when content from different source locations is incorrectly combined.
- Informativeness rating: Aspect-level quality ratings assess whether each aspect receives necessary and sufficient coverage of the reference's information.The informativeness scale ranges from irrelevant summaries to summaries covering key points without unfaithful errors.
- Informativeness rating: Ratings distinguish partial coverage and limited errors from summaries that miss many important topics or contain major fabricated or distorted facts.