PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics

Huan Yee Koh, Jiaxin Ju, Ming Liu, Shirui Pan

arXiv:2207.00939v1cs.CL

TL;DR

Long documents are difficult to summarize because they contain more dispersed information while summaries remain length-constrained, and existing systems face model and evaluation limitations. This paper surveys long document summarization across datasets, models, and metrics, combining literature review with empirical analyses. It concludes by identifying patterns in dataset structure and model mechanisms while outlining future research directions.

  • Problem

    Long documents require greater compression and coverage of dispersed content, while automatic summarization remains challenging and standard evaluation can correlate weakly with human quality judgments.

  • Method

    The paper conducts a comprehensive survey and systematic empirical analysis of benchmark datasets, summarization models, and evaluation metrics for long document summarization.

  • Results

    The survey finds that long-document systems use mechanisms such as discourse bias, efficient attention, and content selection, while retrieve-then-summarize models show mixed multidimensional results.

  • Takeaways & Limitations

    Long document summarization research must account for document structure, long-input processing, content selection, and evaluation dimensions beyond standard lexical overlap.

  • Takeaways & Limitations

    The multidimensional experiment omitted factual-consistency evaluation and was conducted only on the arXiv benchmark dataset, so metric robustness should be interpreted with care.

Abstract

from arXiv · show

Long documents such as academic articles and business reports have been the standard format to detail out important issues and complicated subjects that require extra attention. An automatic summarization system that can effectively condense long documents into short and concise texts to encapsulate the most important information would thus be significant in aiding the reader's comprehension. Recently, with the advent of neural architectures, significant research efforts have been made to advance automatic text summarization systems, and numerous studies on the challenges of extending these systems to the long document domain have emerged. In this survey, we provide a comprehensive overview of the research on long document summarization and a systematic evaluation across the three principal components of its research setting: benchmark datasets, summarization models, and evaluation metrics. For each component, we organize the literature within the context of long document summarization and conduct an empirical analysis to broaden the perspective on current research progress. The empirical analysis includes a study on the intrinsic characteristics of benchmark datasets, a multi-dimensional analysis of summarization models, and a review of the summarization evaluation metrics. Based on the overall findings, we conclude by proposing possible directions for future exploration in this rapidly growing field.

1 INTRODUCTION

Long document summarization addresses the growing difficulty of manually processing lengthy texts and the limitations of existing automatic systems. This survey reviews datasets, models, and metrics, empirically analyzes each component, and identifies future research directions.

  • Motivation: Manual summarization becomes impractical as information growth and document length increase, leaving valuable knowledge unnoticed.The authors frame this challenge as a bottleneck motivating automatic long document summarization research.
  • Motivation: Automatic summarization shortens source texts while preserving their main ideas, but remains challenging and not ready for broad day-to-day use.The process can reduce information-processing time, support faster search, and facilitate learning.
  • Background: Extractive, abstractive, and hybrid approaches represent the three general paradigms for automatic text summarization.Extractive systems copy salient sentences, abstractive systems generate text from salient concepts, and hybrid systems combine extraction with rewriting.
  • Evaluation: ROUGE measures lexical overlap efficiently, but recent findings show weak correlation with human judgments of summary quality.This limitation motivates examining alternative evaluation metrics for long document summarization.
  • Contribution: The survey comprehensively reviews benchmark datasets, summarization models, and evaluation metrics within long document summarization.It addresses a reported lack of comprehensive survey coverage in the field.
  • Contribution: Fine-grained human analysis and ad-hoc experiments examine each component, followed by discussion of limitations and future directions for models, datasets, metrics, and applications.The paper also surveys representative model architectures and organizes its analysis across multiple sections.

2 FUNDAMENTALS OF LONG DOCUMENT SUMMARIZATION

Long documents differ from short documents in length, content breadth, and coherence, creating additional compression and organization challenges. The survey therefore treats long document summarization as a distinct task requiring attention to dispersed narratives and section structure.

  • Fundamentals: The task is characterized by three fundamental aspects: document length, breadth of content, and degree of coherence.These dimensions provide the paper’s conceptual distinction between short and long document summarization.
  • Document Length: In machine learning, a document is long when models designed for normal documents cannot process it effectively because of hardware or model limitations.The definition is operational rather than based only on the document’s lexical token count.
  • Breadth of Content: As documents grow longer, informative non-redundant content increases while reasonable summary length remains constrained, requiring greater source compression.The survey reports that summary-to-source length becomes exponentially shorter as source length increases.
  • Degree of Coherence: Sectional organization makes long documents harder to summarize because combining salient text across sections can affect fluency, redundancy, and semantic coherence.Sections differ in content while remaining connected to a key narrative.
  • Survey Scope: The survey empirically examines benchmark datasets, summarization models, and evaluation metrics in response to these long-document characteristics.Its coverage is organized around the three research components identified in the paper.

3 DATASETS

The survey examines long-document benchmark datasets, their intrinsic characteristics, and their implications for summarization models and evaluation. Compared with short-document benchmarks, long-document datasets are substantially longer, more compressed, less layout-biased, and exhibit dataset-specific abstractiveness and quality issues.

  • Dataset Characteristics: Long-document datasets generally average at least 3,000 source tokens, whereas CNN-DM averages 774 tokens, exceeding vanilla Transformer input limits.Except for BillSum, the long-document datasets require truncating at least half of the source document when models accept 1,024 tokens.
  • Dataset Characteristics: Long-document datasets have token- and sentence-level compression ratios 1.4 and 2.2 times those of short-document datasets, respectively.The higher ratios imply greater information loss, sparser salient content, and/or more redundancy, increasing the task’s difficulty.
  • Dataset Characteristics: Except for BIGPATENT, long-document datasets show greater extractive coverage and density than short-document datasets, supporting extractive modeling in some settings.The authors note that long-document genres often contain domain-specific formulas and terminology, while abstractive systems can produce factual inconsistencies.
  • Dataset Characteristics: Salient content is distributed more uniformly across long documents, so truncating them to leading content can cause significant performance degradation.This contrasts with short-document layout bias, where important sentences often cluster near the beginning.
  • Dataset Comparison: BIGPATENT is identified as the most suitable benchmark for supervised abstractive long-document summarization, but few works evaluate on it.Its suitability is attributed to low coverage and density, many training pairs, and high uniformity in salient content.
  • ArXiv Analysis: More than 60% of annotated ground-truth summaries in the sampled arXiv test set contain noise, exceeding the 54% reported for XSUM.The survey releases annotated data to support analysis of these errors and future benchmark-quality improvements.

4 MODELS

The survey organizes long-document summarization models by extractive, abstractive, and hybrid approaches, then analyzes architectures and mechanisms for adapting models to long inputs. It highlights graph-based ranking for unsupervised extractive systems and pretrained Transformers for supervised abstractive systems as the two standout base architectures.

  • 4.1 Overview: Extractive systems select salient source fragments, abstractive systems paraphrase important content, and hybrid systems rewrite summaries from selected source subsets.The hybrid approach reduces the burden on abstractive models by separating content selection from abstract generation.
  • 4.1 Overview: Models are decomposed into a main architecture and its mechanisms, distinguishing core framework structure from implementation modifications.Different models can share an architecture while using different mechanisms, or use different architectures with similar mechanisms.
  • 4.2 Main Architecture and its Mechanisms: Graph-based extractive models rank sentences by centrality and can incorporate external knowledge, remaining strong baselines for unsupervised long-document summarization.Sentence similarity may use dot product or cosine similarity, with TF-IDF or BERT representations; top-k sentences are extracted by centrality ranking.
  • 4.3 Mechanisms of Transformer-based Architectures: Efficient-attention mechanisms address vanilla Transformers' O(n^2) memory complexity, enabling models such as Longformer to process substantially longer inputs.Longformer supports up to 16,384 tokens when fine-tuning pretrained BART, compared with BART's original 1,024-token limit; BigBird adds a random attention pattern and matches Longformer's ROUGE performance.
  • 4.3 Mechanisms of Transformer-based Architectures: Prompt engineering and signal guidance provide additional inputs or language prompts that help pretrained models identify and summarize important source content.CRTLSum uses extracted keyword prompts with BART and reports significant improvement over vanilla fine-tuned BART on the arXiv dataset; GSum uses dual encoders for documents and extracted signals.
  • 4.4 Summary of Trends in Long Document Summarization Systems: Graph-based ranking and pretrained Transformer architectures can both be adapted from short-document settings to long-document summarization through novel mechanisms.The survey identifies graph-based ranking as prominent for unsupervised extractive models and pretrained Transformers as prominent for supervised abstractive models.

5 MULTI-DIMENSIONAL ANALYSIS OF LONG DOCUMENT SUMMARIZERS

The survey's multi-dimensional analysis experimentally examines graph-based extractive and Transformer-based abstractive architectures and their mechanisms on the arXiv benchmark. The documents average 6,446 tokens, and supervised extractive mechanisms are excluded because their architectures vary substantially.

  • Experimental Setup: The experiment tests graph-based extractive and Transformer-based abstractive architectures and their mechanisms on the arXiv benchmark dataset.The design aims to understand the reasons behind the popularity of these architectures and mechanisms.
  • Experimental Setup: 6,446 tokens is the average document length in the arXiv dataset used for the experiment.The experiment does not examine supervised extractive mechanisms because the architectures used across proposed models vary greatly.

5.1 Implementation

The experiments compare unsupervised graph-based extractive models and supervised abstractive Transformer models, varying sentence encoding, discourse bias, attention, input length, and content selection. Outputs are evaluated across relevance, informativeness, and semantic coherence using multiple metrics.

  • Graph - Unsupervised Extractive: Four unsupervised graph models vary sentence encoders between Tf-Idf and BERT SentenceTransformer, with or without long-document discourse bias.Discourse bias incorporates positional and sectional information into sentence centrality scoring.
  • Transformer - Supervised Abstractive: BART experiments compare truncated full self-attention, Longformer attention up to 4,096 or 16,384 tokens, and retrieve-then-summarize variants using selected subsets.The retrieve-then-summarize variants process subsets capped at 1,024 or 4,096 tokens, depending on attention type.
  • Transformer - Supervised Abstractive: PEGASUS experiments compare the original 1,024-token model with a BigBird efficient-attention model supporting 4,096 tokens.Both pretrained arXiv model variants were obtained from the original authors.
  • Assessing Model Outputs: Model outputs are assessed across relevance, informativeness, and semantic coherence rather than relying only on ROUGE.ROUGE and BERTScore measure relevance; section coverage approximates informativeness; BERT next-sentence prediction approximates semantic coherence.
  • Implementation: All models use the same arXiv split of 203,037/6,436/6,440 for training, validation, and testing, with consistent preprocessing and shared GPU conditions.Fine-tuned models use an RTX 3090 GPU with 24 GiB of memory, and ROUGE is computed with pyrouge.

5.2 Results and Analysis

The experiments find that discourse-aware and semantically informed extractive models improve relevance and informativeness, while longer Transformer inputs improve all evaluated dimensions. Retrieve-then-summarize models achieve strong ROUGE results but can lose semantic coherence.

  • Graph - Unsupervised Extractive: BERT sentence encoding boosts unsupervised summarization performance in relevance and informativeness by capturing semantically important sentences despite wording differences.Without semantically rich representations, extracted summaries may contain more redundancy.
  • Graph - Unsupervised Extractive: Discourse bias greatly improves extractive relevance and informativeness, but combining sentences from different sections can reduce semantic coherence.The mechanism adds positional and sectional bias to sentence centrality scoring.
  • Transformer - Supervised Abstractive: PEGASUS-only models outperform BART-only models across relevance, informativeness, and semantic coherence, although the source of the advantage is inconclusive.The gain may reflect the GSG pretraining task, the pretraining corpus, or both, and becomes less obvious with 4,096-token efficient attention.
  • Transformer - Supervised Abstractive: Retrieve-then-summarize BART models achieve state-of-the-art ROUGE results and generally improve across dimensions when processing longer retrieved subsets, but perform worst in semantic coherence.The authors hypothesize that inference-time disconnects between content selection and encoder-decoder modeling contribute to incoherence.
  • Transformer - Supervised Abstractive: BART informativeness is 10 points lower at 1,024 than at 16,384 input tokens, while longer inputs improve relevance, informativeness, and semantic coherence overall.ROUGE does not fully capture this performance difference.
  • Overall Findings: Across the ad-hoc experiments, explicit discourse structure and longer Transformer inputs yield promising outcomes, whereas retrieve-then-summarize systems can trade ROUGE strength for coherence.The analysis systematically examines common long-document summarization approaches.

5.3 Limitation of Experiment

The multi-dimensional experiment omits factual-consistency evaluation because available metrics have important limitations. Its metric analysis should therefore be interpreted cautiously, especially given its single-dataset scope.

  • Metric Limitations: Factual consistency is not evaluated because existing metrics have input-length limits, implementation difficulty, and performance variation across benchmarks.The tested textual-entailment adaptations had almost no discriminative ability.
  • Scope Boundary: The robustness of the relevance, informativeness, and semantic-coherence metrics should be interpreted with care.The experiment was conducted only on arXiv because it was the sole dataset with all required pretrained weights publicly available.

6 METRICS

The survey reviews relevance, factual consistency, conciseness, and semantic coherence metrics, emphasizing that long-document evaluation requires practical, multidimensional assessment beyond lexical overlap.

  • Relevance: ROUGE measures exact lexical overlap but can penalize paraphrases and ignores factual consistency, fluency, and conciseness.Its common variants are ROUGE-1, ROUGE-2, and ROUGE-L.
  • Relevance: Soft-overlap metrics compare contextual representations rather than exact lexical matches, but evaluator quality may vary across domains.The survey questions whether BERT-base reliably discriminates summaries with and without errors across domains.
  • Relevance: Reference-free relevance metrics score candidate summaries against pseudo-references generated from source documents, with some requiring supervised high-quality summaries.Other reference-free approaches can generate scores without high-quality summaries as supervisory signals.
  • Factual Consistency: Factual-consistency metrics use entailment classification or question answering to verify summary content against the source document.Question-answering methods compare answers conditioned on the source text and candidate summary; QAGen generates questions and answers concurrently.
  • Factual Consistency: Factual-consistency evaluation can be multidimensional, because metric effectiveness depends strongly on the evaluated architecture and benchmark dataset.FRANK distinguishes semantic, discourse, and content-verifiability errors.
  • Research Efforts on Metrics in the Long Document Domain: Long-document evaluation remains constrained by adaptation difficulty and computation costs, limiting adoption of proposed factual-consistency metrics.A competing metric required approximately four days to evaluate CNN-DM and would likely take significantly longer on long-document benchmarks.

7 APPLICATIONS

Long-document summarization is expanding into research, industrial, and commercial settings, with applications extending beyond summarization itself.

  • Applications: Long-document summarization models have been explored for scientific research and industrial use, including news, business, and consulting services.The field has progressed from scientific-paper benchmarks toward commercial Software-as-a-Service products.
  • Applications: Successful long-document summarization can support auxiliary tasks such as video captioning, long-document question answering, and multimodal tasks.The task is framed as identifying important aspects from long sequences, which connects it to several NLP applications.

8 GENERAL CHALLENGES AND FUTURE DIRECTIONS

The survey identifies unresolved challenges in model understanding, document structure, input length, benchmark quality, domain coverage, and metric practicality, and proposes corresponding research directions.

  • Neural Models and Long Sequence Reasoning: Efficient Transformers can improve performance across NLP tasks, but the reasons for these gains and their domain dependence remain poorly understood.The survey calls for research into how efficient attention and pretraining contribute to long-sequence reasoning.
  • Neural Models and Long Sequence Reasoning: Transformer summarizers still make similar amounts of discourse-related errors as RNN models, while studies report conflicting findings about efficient attention patterns.Learnable attention patterns outperform fixed local patterns in one study, whereas another finds larger windows do not change attended-neighbor distance.
  • Document Structure: Section-based discourse information can improve an unsupervised graph model, but such information may be unavailable or unreliable in practice.This limits models that require explicit section-based discourse information.
  • Long-Document Input Length: Documents exceeding hundreds of thousands of tokens cannot be summarized by standard models without long-document adaptation because current input limits truncate source text.The survey presents more practical directions for handling these extremely long documents.
  • Benchmark Datasets: 60% of arXiv reference summaries contain some errors, including 15% with significant errors affecting at least half the summary.The survey calls for higher-quality benchmark datasets and broader coverage beyond legislative and scientific domains.
  • Evaluation Metrics: Proposed evaluation improvements often lack wide practicality, with one factual-consistency metric requiring more than four days on a single GPU for one test set.The survey highlights availability and computational cost as barriers to adoption.

9 CONCLUSION

The survey synthesizes long-document datasets, models, and evaluation metrics, combining literature review with empirical analyses to characterize the field and identify future needs.

  • Datasets: Long documents require greater source compression and identification of key narratives, which are more uniformly scattered across the source.They are also often more extractive and contain explicit discourse structures that models can exploit.
  • Models: The survey reviews and compares architectures and mechanisms for generating long-document summaries, then evaluates widely used designs through ad-hoc experiments.Its model analysis spans architectures and mechanisms used across prior work.
  • Evaluation Metrics: Evaluation research has advanced, but metrics that are readily applicable to long-document summarization remain insufficiently studied.The survey calls attention to practicality as a continuing gap in evaluation.
  • Future Directions: The survey concludes by examining applications and proposing future exploration for this rapidly growing research area.Its overall analysis covers benchmark datasets, summarization models, and evaluation metrics.

10 SUPPLEMENTARY MATERIALS

The supplementary materials document the survey’s long-document system comparisons, graph-based sentence ranking, metric adaptations, and implementation details. They also describe textual entailment adaptation for evaluating factual consistency.

  • Long Document Summarization Systems: Table 4 summarizes long-document baseline and state-of-the-art systems, including whether they use discourse-structure priors and how much truncation occurs.The Prior column records inductive bias use, while Trunc records significant truncation across five benchmarks.
  • Graph-based Ranking Algorithm in Experimental Section: Graph-based ranking computes sentence centrality from bias-adjusted similarities between sentences and, where applicable, sections.Sentence representations use an encoder; section nodes represent averages of sentences within the section.
  • Graph-based Ranking Algorithm in Experimental Section: The supplementary implementation compares Tf-Idf and BERT sentence encoders with long-document discourse bias, alongside encoder-specific baselines.The discourse bias incorporates intra-section and inter-section importance, while BERT replaces the Tf-Idf encoder in one variant.
  • Graph-based Ranking Algorithm in Experimental Section: Hyperparameter tuning varies section-position and intra- versus inter-section weighting, with λ1=0.5 and λ2=1 used in the experiments.The original method pruned less important sentences with λ1=0, whereas this implementation downweights them using λ1=0.5.
  • Textual Entailment as Factual Consistency Metric: Table 5 reports entailment, contradiction, and neutrality results separately for high-quality and low-quality annotated arXiv reference summaries.The Good and Bad columns distinguish the two annotation groups after long-document adaptation.
  • Textual Entailment as Factual Consistency Metric: Long-document textual entailment evaluation adapts a 512-token-limit BERT model by adding content selection for thousands-token documents.The entailment task classifies summary sentences as entailment, neutral, or contradiction; ideal summaries should avoid contradiction.

I. ACL

This section lists metric-related summarization papers in ACL main conferences through September 2021. It also records that EACL, CoNLL, and AACL had no metric-related summarization research papers.

  • I. ACL: The surveyed metric-related literature includes papers on scoring ranges, facet-aware evaluation, question answering, faithfulness, factuality, and centrality-weighted relevance.These topics span reference-based and reference-free approaches to summarization evaluation.
  • I. ACL: EACL, CoNLL, and AACL have no metric-related summarization research papers in the listed conference coverage.
Loading 2207.00939v1…