PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models

Haopeng Zhang, Philip S. Yu, Jiawei Zhang

arXiv:2406.11289v1cs.CL

TL;DR

Text summarization research has progressed through statistical, deep-learning, PLM, and LLM paradigms, but evaluation quality and factual reliability remain important challenges. This survey synthesizes pre-LLM methods and resources with LLM-era benchmarking, modeling, and evaluation studies, and discusses trends, open challenges, and future directions.

  • Problem

    Existing surveys provide limited comprehensive coverage of summarization’s rapid progression into the PLM and LLM eras, while automatic evaluation still correlates only limitedly with human judgment.

  • Method

    The survey organizes summarization research across paradigm shifts, reviewing datasets, metrics, pre-LLM methods, LLM-era benchmarking, modeling, and evaluation studies.

  • Results

    The synthesis captures research evolution across pre-LLM and LLM eras and identifies trends, open challenges, and prospective research directions.

  • Takeaways & Limitations

    The survey provides a consolidated resource for understanding advancements and future prospects in the evolving summarization field.

  • Takeaways & Limitations

    LLM summarization remains constrained by hallucination and bias, including factually incorrect content and biased summaries.

Abstract

from arXiv · show

Text summarization research has undergone several significant transformations with the advent of deep neural networks, pre-trained language models (PLMs), and recent large language models (LLMs). This survey thus provides a comprehensive review of the research progress and evolution in text summarization through the lens of these paradigm shifts. It is organized into two main parts: (1) a detailed overview of datasets, evaluation metrics, and summarization methods before the LLM era, encompassing traditional statistical methods, deep learning approaches, and PLM fine-tuning techniques, and (2) the first detailed examination of recent advancements in benchmarking, modeling, and evaluating summarization in the LLM era. By synthesizing existing literature and presenting a cohesive overview, this survey also discusses research trends, open challenges, and proposes promising research directions in summarization, aiming to guide researchers through the evolving landscape of summarization research.

1 INTRODUCTION

Text summarization has evolved from statistical and deep-learning systems through PLM fine-tuning into an LLM era that creates both new opportunities and research challenges. This survey synthesizes methods, evaluation, datasets, trends, open challenges, and future directions across these paradigm shifts.

  • Summarization research progressed from unsupervised statistical methods to supervised deep learning, PLM fine-tuning, and LLM-based approaches.The survey frames this development as four major stages.
  • LLMs have transformed summarization through extensive pretrained knowledge and strong text understanding, analysis, and generation capabilities.The paper describes LLMs as enabling a new era for summarization research and products.
  • The survey reviews pre-LLM methods, datasets, evaluation metrics, and representative LLM-era benchmarking, modeling, and evaluation studies.It organizes the review around both historical paradigms and current LLM research.
  • It also analyzes research trends and open challenges while proposing future directions for summarization research.The stated aim is to support understanding of ongoing developments and prospects.
  • Existing surveys largely emphasized statistical and deep-learning approaches, leaving comprehensive coverage of PLM and LLM-era progress scarce.The survey identifies this lack of up-to-date synthesis as an ongoing obstacle to progress.

2 BACKGROUND

The background section introduces text summarization through its task categorization, problem formulations, evaluation metrics, and benchmark datasets.

  • The section covers task categorizations, extractive and abstractive formulations, pre-LLM evaluation metrics, and widely used benchmark datasets.

2.1 Categorization

Summarization approaches are categorized by input format, output style, and underlying paradigm. Extractive, abstractive, and hybrid methods differ in how they select or generate content, while research has progressed through four major paradigm stages.

  • Input format: Summarization inputs include single documents, document clusters, and queries, corresponding to SDS, MDS, and QFS.QFS produces summaries addressing an input topic, keywords, or entity.
  • Output style: Extractive methods select salient spans or sentences from the source, whereas abstractive methods generate summaries from scratch with novel content.Extractive systems are generally faithful and accurate but may be redundant or incoherent; abstractive systems are more flexible but risk hallucination and unfaithfulness.
  • Output style: Hybrid methods extract key information first and use it to guide subsequent abstractive generation.They aim to combine strengths of extraction and abstraction.
  • Output style: Figure 3 contrasts extractive and abstractive summaries for a CNN/DailyMail news article.
  • Paradigm evolution: The statistical stage used unsupervised heuristic, optimization, and graph methods with hand-crafted or frequency-based features such as TF-IDF.
  • Paradigm evolution: The deep learning stage trained neural networks supervised on document-summary pairs and introduced training corpora for summarization.
  • Paradigm evolution: The PLM fine-tuning stage used self-supervised pretrained models in a pre-train, then fine-tune pipeline, improving language understanding and generation.
  • Paradigm evolution: The LLM stage enabled zero-shot and few-shot summarization through stronger language understanding and instruction-following capabilities.

2.2 Problem Formulation

Extractive summarization selects salient source sentences, while abstractive summarization generates summaries autoregressively from encoded document representations. Hybrid systems add extracted information or other guiding signals to the generation process.

  • 2.2.1 Extractive Summarization Formulation.: Extractive summarization forms a summary by directly selecting key sentences from a document.
  • 2.2.1 Extractive Summarization Formulation.: Given n document sentences, an extractive system constructs a shorter summary of m sentences where m ≪ n.
  • 2.2.1 Extractive Summarization Formulation.: Most extractive systems formulate sentence selection as binary sequence labeling, with label 1 selecting salient sentences and label 0 excluding them.
  • 2.2.1 Extractive Summarization Formulation.: Because sentence-level extractive gold labels are rarely available, greedy ROUGE-2 maximization is commonly used to construct a sub-optimal oracle.Sentences are added incrementally to maximize overlap with the reference summary.
  • 2.2.2 Abstractive Summarization Formulation.: Abstractive summarization uses an encoder-decoder architecture in which the encoder represents the source and the decoder generates the summary word by word.Most systems use autoregressive sequential decoding and teacher forcing.
  • 2.2.2 Abstractive Summarization Formulation.: Abstractive models train parameters to maximize the conditional likelihood of generated summary words given the source document.
  • 2.2.3 Hybrid.: Hybrid systems augment abstractive generation with an extracted information signal G alongside the source document.Guiding signals can include extracted summaries, keywords, relations, or externally retrieved information.

2.3 Evaluation Metrics

Summary evaluation has progressed from reference-based lexical overlap toward contextual, generation-based, factual-consistency, coherence, and redundancy measures. However, automatic metrics remain shaped by reference similarity and model-dependent assumptions.

  • Automatic summary quality is commonly assessed along coherence, consistency, fluency, and relevance.
  • Similarity-based evaluation: ROUGE-1, ROUGE-2, and ROUGE-L measure unigram overlap, bigram overlap, and longest common sequence with a reference summary.These scores are widely used to indicate informativeness and fluency, respectively.
  • Similarity-based evaluation: Because ROUGE relies on exact n-gram matches, it overlooks synonymous expressions and penalizes novel wording.
  • Similarity-based evaluation: BERTScore, MoverScore, and Sentence Mover’s Similarity use contextualized embeddings but may inherit encoder biases and have low interpretability.
  • Factual consistency: Factual-consistency metrics use entailment or question answering to compare generated summaries with their source documents.Entailment methods include FactCC, DAE, and SummaC, while QA-based methods include FEQA, QAGS, and Questeval.
  • Other evaluation dimensions: SNaC evaluates narrative coherence in long summaries, while unique n-gram ratios and entropy-based diversity measures target redundancy.

2.4 Summarization Datasets

Summarization datasets span news, informal text, dialogue, scientific writing, legal documents, reviews, meetings, and multilingual content. They differ in domain, format, scale, language, document length, and the number or type of reference summaries.

  • Public datasets have driven recent summarization progress and vary across domain, format, size, and number of gold summaries.
  • Single-document datasets: CNN/DM is a widely used news dataset containing articles and journalist-written highlights from CNN and the DailyMail.
  • Single-document datasets: XSum contains one-sentence BBC article summaries, but some references include information not directly inferable from their source documents.
  • Specialized datasets: Specialized datasets cover knowledge articles, informal stories, dialogues, emails, scientific publications, patents, legislation, financial reports, and multilingual news.Examples include WikiHow, Reddit, SAMSum, AESLC, PubMed, arXiv, BIGPATENT, BillSum, FINDSum, and XL-Sum.
  • Multi-document datasets: Multi-document datasets use news, web, scientific, review, and event collections, including DUC, MultiNews, WikiSum, WCEP, Multi-XScience, and Yelp.
  • Query- and aspect-focused datasets: Query- and aspect-focused datasets provide summaries conditioned on information needs, such as meeting queries in QMSum and topic-specific summaries in NewTS.TD-QFS targets medical texts with keyword queries and substantial query-irrelevant information.

3 SUMMARIZATION METHODS PRIOR TO LLMS

Before the LLM era, summarization research developed through three major paradigms: statistical methods, deep learning-based approaches, and PLM-based fine-tuning.

  • The pre-LLM literature is organized into statistical methods, deep learning-based approaches, and PLM-based fine-tuning approaches.

3.1 Statistical Summarization Methods

Early summarization systems primarily performed extractive selection using statistical features, optimization, or graph-based ranking. These methods evolved from heuristic sentence scoring toward globally constrained selection and graph centrality, with later neural representations improving graph similarity computation.

  • Early extractive systems used TF-IDF and handcrafted features, with representative heuristic, optimization-based, and graph-based methods.
  • Heuristic-based Methods: Luhn scored sentences by content-word frequency and extracted high-scoring sentences for literature abstracts.
  • Heuristic-based Methods: Edmundson combined word frequency, sentence position, title or heading matches, and cue-word frequency in a linear sentence-ranking model.
  • Heuristic-based Methods: MMR greedily balances sentence relevance and information novelty to reduce redundancy during extractive selection.
  • Optimization-based Methods: Optimization methods select sentences to maximize important-content coverage while minimizing redundancy under a length constraint.Formulations include maximum coverage and tree knapsack problems, alongside ILP, DPP, and submodular approaches.
  • Graph-based Methods: LexRank and TextRank represent documents as graphs and identify central sentence nodes using PageRank-inspired ranking.Nodes represent sentences, edges represent sentence similarity, and low-weight edges may be pruned before graph computation.
  • Graph-based Methods: PACSUM extends TextRank with directed edges, positional effects on centrality, and neural representations for sentence similarity.

3.2 Deep Learning Summarization Methods

Deep learning methods expanded summarization beyond feature-based systems by learning text representations from data and supporting extractive, abstractive, reinforcement-learning, and graph-based approaches. Research explored CNNs, RNNs, reinforcement learning, and graph neural networks to model salience, generation, structure, and training objectives.

  • Word embeddings enabled deep models to learn continuous sentence and document representations directly from data, reducing reliance on human-engineered features.
  • CNN-based methods: CNN-based methods supported both extractive selection through dense sentence representations and abstractive summarization through sequence-to-sequence architectures.
  • RNN-based methods: RNN-based systems modeled documents hierarchically for extractive salience prediction and used attention, copying, or hierarchical structure for abstractive generation.
  • Reinforcement learning: Reinforcement learning addressed exposure bias and objective mismatch by combining likelihood-based training with reward-driven optimization.
  • Graph neural networks: Graph neural networks represented documents with sentence nodes and similarity or discourse edges, aggregating structural information to predict salient sentences.Graph variants included discourse graphs, multiplex graphs for multiple relationship types, and hypergraphs for higher-order cross-sentence relations.
  • Hybrid and emerging methods: Pointer-generator and diffusion-based methods further combined extraction and generation by copying source content, selecting output words, or matching generated sentence representations.

3.3 Pre-trained Language Model Summarization Methods

Pre-trained language models substantially advanced summarization by transferring self-supervised representations into extractive and abstractive systems. The surveyed methods span encoder-only, encoder-decoder, and decoder-only frameworks, with research addressing faithfulness, long documents, multi-document inputs, and task-specific pre-training.

  • Transformer self-attention and large-scale self-supervised pre-training improved summarization systems by enabling efficient learning from unlabeled corpora.
  • Pre-training architectures: Encoder-only, encoder-decoder, and decoder-only frameworks differ in whether they encode bidirectionally, generate autoregressively, or combine both operations.Encoder-only models mask tokens, decoder-only models predict from leftward context, and encoder-decoder models reconstruct corrupted inputs before generation.
  • Representative PLMs: BERT became a widely adopted encoder-only backbone, while T5 and BART supported the growing focus on abstractive summarization with encoder-decoder architectures.
  • Encoder-only methods: PreSum demonstrated a unified BERT-based framework for extractive and abstractive summarization using inter-sentence Transformer layers and differentiated encoder-decoder fine-tuning.
  • Fine-tuning methods: Fine-tuning research targeted faithfulness, controllability, exposure bias, representational collapse, and error correction through regularization, guidance, contrastive learning, and post-processing.
  • Task-specific pre-training: Task-specific pre-training produced models for extractive, abstractive, long-document, and multi-document summarization, including hierarchical, gap-sentence, local-global attention, and entity-pyramid objectives.LED scales linearly with sequence length using local windowed and task-motivated global attention, while PRIMERA uses entity pyramid masking for multi-document representation.

4 LLM-BASED SUMMARIZATION RESEARCH

LLM summarization research spans broad benchmarking and specialized applications, revealing strong capabilities alongside persistent factuality, controllability, positional, resource, and domain-specific limitations.

  • Benchmarking studies: Human evaluators preferred GPT-3 summaries over fine-tuned models despite lower reference-based and reference-free automatic scores.This divergence highlights limitations in using automatic metrics alone for LLM summarization.
  • Benchmarking studies: LLM-generated summaries were judged comparable to human-written summaries, with instruction tuning improving zero-shot performance and some evaluations finding better factual consistency.Other human evaluations also reported fewer extrinsic hallucinations in LLM-generated summaries.
  • Generic summarization: ChatGPT underperformed supervised systems on ROUGE for extractive summarization but achieved higher scores on LLM-specific evaluation metrics.The study also examined in-context learning and an extract-then-generate pipeline.
  • Controllability and specialized tasks: LLMs show varied task behavior: interactive prompting improves multilingual informativeness-conciseness balance, while limited coverage remains challenging for multi-document summarization.Instruction-controllable summarization and audience- or style-adaptive generation also remain difficult, with occasional factual errors and hallucinations.
  • Faithfulness and factuality: LLMs often fail to detect factual inconsistencies, including copied errors, and can perform near random chance as binary factual evaluators.Benchmarks include human-verified factual summaries, inconsistent summaries, and detailed explanations for dialogue summaries.
  • Applications and characteristics: LLMs exhibit fairness, positional, medical, biomedical, and code-summarization limitations despite strong results in some specialized settings.Observed issues include low fairness, U-shaped context utilization, medical hallucinations, poor multi-document aggregation, and code performance that varies by representation and prompting strategy.
  • Deployment challenges: Resource demands and privacy concerns limit widespread LLM use, motivating compact local models that retain summarization capabilities.These constraints are especially relevant when sensitive data are processed through LLM-as-a-service APIs.

5 OPEN PROBLEMS AND FUTURE DIRECTIONS

The survey identifies open challenges and future directions for LLM-based summarization, including hallucination, bias, efficiency, personalization, interpretability, domain adaptation, and emerging multimodal and human-centered applications.

  • Research trends: LLM-based summarization remains nascent, with most research focused on benchmarking rather than developing methods that better exploit these models.Open-source LLMs are enabling more modeling research, but effective systems for diverse scenarios remain a future focus.
  • Future directions: Future summarization research is expanding toward multimodal, personalized, human-in-the-loop, real-time, sentiment-aware, and diversified tasks across broader domains.Applications increasingly include finance, law, healthcare, education, images, tables, source code, audio, and video.
  • Research trends: Human evaluation and human-in-the-loop systems are increasingly important because automatic metrics may not capture summary quality or user needs.Hybrid approaches combine human expertise with LLM capabilities to improve summary quality and reliability.
  • Open challenges: Hallucination remains a major deployment barrier, requiring stronger measurement and mitigation to achieve faithful and factual summaries.Proposed mitigation strategies include enhanced training, retrieval augmentation, self-reflection, and post-processing.
  • Open challenges: LLM summarization must address biased outputs, high computational demands, personalization, and limited interpretability, especially in sensitive domains.These challenges concern fairness, scalability, privacy, transparency, and users’ ability to understand why information was included or omitted.
  • Future directions: Domain-specific adaptation and ethical AI practices are needed for trustworthy deployment because zero-shot generalization can lack specialized knowledge and terminology.The survey emphasizes transparency, fairness, bias mitigation, controlled incorporation of updated knowledge, and adaptation of open-source LLMs.

6 CONCLUSION

The survey traces profound advances in text summarization driven by deep neural networks, PLMs, and LLMs, while examining both pre-LLM and LLM-era research. It also identifies research trends, open challenges, and future directions for advancing summarization techniques.

  • The survey details the evolution of text summarization across deep neural networks, PLMs, and LLMs.
  • It comprehensively examines summarization research in both the pre-LLM and LLM eras.
  • The survey highlights research trends and delineates open challenges in summarization.
  • It proposes future research directions crucial for advancing summarization techniques.
Loading 2406.11289v1…