PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods

Yang Zhang, Hanlei Jin, Dan Meng, Jun Wang, Jinghua Tan

arXiv:2403.02901v3cs.AI

TL;DR

Existing ATS surveys often use theoretical categorizations, while the field has shifted with the emergence of LLMs and their flexible summarization capabilities. This paper surveys ATS from a process-oriented perspective, reviews recent LLM-based methods, and introduces an automated retrieval approach. Its retrieval algorithm achieved strong classification performance across binary and multi-class annotation tasks, while the survey identifies domain coverage and hallucination as continuing boundaries.

  • Problem

    Earlier ATS surveys generally lacked practical, process-oriented coverage, while high-quality datasets for specialized domains and factual grounding remain limited.

  • Method

    The paper provides a comprehensive ATS survey, analyzes LLM-based methods, and uses an automated retrieval algorithm combining keyword searches with LLM prompting to collect papers.

  • Results

    The retrieval algorithm achieved 94.2% accuracy for is_LLM and 93.4% for is_dataset, with robust performance across Type, Methodology, and Domain classification tasks.

  • Takeaways & Limitations

    The survey offers an up-to-date account of conventional and LLM-based ATS, including how in-context learning, prompt engineering, and few-shot learning have reshaped the field.

  • Takeaways & Limitations

    Current summarization datasets are predominantly news-focused and lack high-quality coverage for specialized domains, while LLM summaries can contain hallucinated or unsupported claims.

Abstract

from arXiv · show

Automatic Text Summarization (ATS), utilizing Natural Language Processing (NLP) algorithms, aims to create concise and accurate summaries, thereby significantly reducing the human effort required in processing large volumes of text. ATS has drawn considerable interest in both academic and industrial circles. Many studies have been conducted in the past to survey ATS methods; however, they generally lack practicality for real-world implementations, as they often categorize previous methods from a theoretical standpoint. Moreover, the advent of Large Language Models (LLMs) has altered conventional ATS methods. In this survey, we aim to 1) provide a comprehensive overview of ATS from a ``Process-Oriented Schema'' perspective, which is best aligned with real-world implementations; 2) comprehensively review the latest LLM-based ATS works; and 3) deliver an up-to-date survey of ATS, bridging the two-year gap in the literature. To the best of our knowledge, this is the first survey to specifically investigate LLM-based ATS methods.

1. Introduction

Automatic Text Summarization addresses the growing difficulty of processing large volumes of text, while conventional approaches remain constrained by fixed paradigms. This survey reviews ATS comprehensively, emphasizes LLM-based methods, and introduces a retrieval algorithm for collecting ATS papers.

  • Rapid growth of textual data has exceeded individuals’ ability to search, read, and process relevant information, motivating ATS.ATS uses NLP, Information Retrieval, Machine Learning, Deep Learning, and LLM techniques to help users grasp key ideas efficiently.
  • Conventional ATS primarily developed around extractive and abstractive paradigms, but abstractive systems often struggled with coherence and factual accuracy.Extractive methods select source sentences or phrases, whereas abstractive methods generate paraphrased sentences.
  • LLMs provide a paradigm-flexible approach that can switch among extractive, abstractive, and hybrid summarization with in-context and few-shot learning.This flexibility requires minimal supervision and does not require retraining or significant architectural changes.
  • The paper develops an automated retrieval algorithm combining keyword searches with LLM prompting to collect and organize ATS research papers.The algorithm is designed to streamline paper collection and can be adapted to other fields.
  • The survey contributes an up-to-date review of conventional and LLM-based ATS approaches, with detailed coverage of in-context learning, prompt engineering, and few-shot learning.It addresses the limited coverage of recent LLM-based ATS methods in earlier surveys.

2. Background of Automatic Text Summarization

ATS evolved from statistical and machine-learning systems to neural and Transformer-based methods, followed by LLMs that flexibly support multiple summarization paradigms. The background classifies conventional and LLM-based approaches and outlines four major LLM-based method families.

  • Early Statistical Era: Early statistical ATS used surface features such as word frequency and sentence position, while later methods incorporated TF-IDF, LSA, BM25, and sentence-relationship maps.These approaches often struggled to capture deeper context and produce coherent summaries for semantically rich text.
  • Machine Learning and Early Neural Networks Era: Machine-learning ATS reduced reliance on manually defined rules through supervised, unsupervised, and semi-supervised learning methods.Examples included SVM, Naïve Bayes Classification, and CRF for sentence classification.
  • Deep Learning and Transformer Era: RNN-based deep-learning models improved language understanding but faced difficulty maintaining coherence over long sequences, limiting long-document summarization quality.
  • Large Language Model Era: LLMs can adapt between extractive and abstractive paradigms or combine both, and their summaries tend to show stronger coherence, fluency, and writing quality than traditional methods.Their few-shot and zero-shot capabilities support summarization with minimal or no task-specific examples.
  • LLM-Based Methods: LLM-based ATS methods are categorized into prompt engineering, retrieval-augmented generation, fine-tuning, and knowledge distillation.These approaches respectively guide generation, add retrieved documents, specialize model parameters, or compress capabilities into more efficient models.
  • Categorization of Automatic Text Summarization: The survey groups ATS into conventional methods—extractive, abstractive, and hybrid—and LLM-based methods that flexibly produce extractive or abstractive outcomes through prompting.

3. Definition and Literature Collection Methodology for Automatic Text Summarization

This section formalizes ATS as a process that transforms input text into a shorter, semantically and syntactically coherent summary, then presents a process-oriented framework and retrieval methodology. It contrasts conventional staged systems with LLM-based end-to-end approaches and evaluates the proposed paper-collection algorithm.

  • ATS Definition: ATS maps input tokens to a substantially shorter summary while minimizing information loss and preserving semantic and syntactic integrity.Extractive summaries select a subset of input tokens, whereas abstractive summaries generate a new sequence.
  • ATS Process: The ATS process is represented as a sequence of data preprocessing, modeling, generation, and evaluation operations.Preprocessing transforms the input, modeling produces a compressed representation, generation produces the summary, and evaluation compares it with a ground truth.
  • Process-Oriented Framework: Conventional ATS uses fragmented stages and manual intervention, whereas LLMs provide unified end-to-end processing that bypasses explicit intermediate steps.The framework identifies autonomous closed-loop operation as the central distinction, with LLMs integrating data construction, modeling, and evaluation.
  • Literature Collection Methodology: The retrieval methodology combines synonym-enhanced keyword searches, title-and-abstract analysis, web crawling, LLM filtering, categorization, and manual review.The three-stage procedure automates collection, filtering, and categorization to address diverse terminology and papers whose relevance appears only in abstracts.
  • Evaluation Results: 94.2% accuracy for is_LLM and 93.4% for is_dataset demonstrate strong binary classification in the retrieval-algorithm evaluation.These tasks identify whether papers concern LLM-based methods or ATS datasets.
  • Evaluation Results: Multi-class performance was robust but weaker for less-represented categories, with macro-F1 below micro-F1 for the Type and Domain tasks.The reported Type scores were 85.6% accuracy, 71.9 macro-F1, and 85.6 micro-F1; Domain scores were 92.0% accuracy, 83.6 macro-F1, and 90.0 micro-F1.

4. ATS Datasets

ATS datasets include established open-source benchmarks and newer or domain-specific resources, but current collections remain concentrated in news and often lack suitability for specialized applications.

  • Dataset Types: ATS datasets are grouped into open-source resources and techniques for building new datasets.The survey distinguishes commonly used datasets from methods for constructing custom resources.
  • Custom Dataset Construction: LLMs can support scalable generation of domain-specific datasets, reducing manual effort and increasing dataset diversity.This approach is proposed to address the limited suitability of general open-source datasets for specialized applications.
  • Dataset Categories: Conventional datasets have over 100 citations and serve as foundational benchmarks, while supplementary datasets have fewer than 100 citations and include newer or domain-specific resources.The two groups are summarized separately by attributes such as size, domain, language, and accessibility.
  • Conventional Datasets: Conventional benchmarks span news, dialogue, procedural tasks, patents, social media, and academic documents.Examples include Gigaword, DUC, CNN & Daily Mail, LCSTS, ArXiv, PubMed, NEWSROOM, WikiHow, Multi-News, SAMSum, and BIGPATENT.
  • Coverage Gaps: Current datasets are predominantly open source and accessible, but they primarily focus on news, leaving high-quality resources for specialized domains underrepresented.The survey identifies financial earnings releases as one example of a domain needing better datasets.

5. Text Pre-processing

Text pre-processing transforms collected raw text into structured data through noise removal, linguistic annotation, normalization, segmentation, and tokenization, with the required operations depending on the model.

  • Overview: Pre-processing follows data collection and transforms raw text into structured-format data.The survey presents it as a necessary stage supported by common methods and tools.
  • Cleaning: Noise removal eliminates HTML tags, extra spaces, blank lines, unknown symbols, gibberish, and, in earlier systems, stop words.Earlier stop-word removal used manually designed lists because these words were considered unhelpful for analysis.
  • Linguistic Processing: Part-of-speech tagging assigns syntactic categories such as verbs, nouns, and adjectives to words.These tags help identify words according to their syntactic roles and sentence context.
  • Normalization: Stemming reduces related words to a basic root form by removing variable endings such as “es” and “ed.”The process was used to reduce computation time and improve recall in information retrieval.
  • Segmentation and Tokenization: Sentence segmentation divides text into sentences, while tokenization divides words into subwords using methods such as BPE or WordPiece.Segmentation may require heuristics for abbreviations, and subword methods construct units suited to model training.
  • Model Dependence: Pre-processing requirements differ across models, although technological advances have reduced its labor intensity without eliminating the need for it.Python tooling includes NLTK, spaCy, and TextBlob for tasks such as tokenization, stemming, tagging, and parsing.

6. Conventional ATS Methods

Conventional ATS methods comprise extractive, abstractive, and hybrid paradigms, each offering distinct trade-offs in precision, flexibility, resource demands, coherence, and compression.

  • Taxonomy: Conventional ATS is classified into extractive, abstractive, and hybrid approaches according to their generation paradigms.Extractive systems select source text, abstractive systems generate new sentences, and hybrid systems combine both techniques.
  • Extractive Summarization: Extractive summarization ranks sentence importance and selects the top-k sentences, using unsupervised or supervised methods.Statistical, clustering, topic-based, machine-learning, and pre-trained models are among the approaches reviewed.
  • Extractive Summarization: Extractive models are accurate, cost-effective, efficient, and effective with limited training data, but can produce redundancy, excessive length, and contextual inconsistencies.Their outputs may preserve terminology precisely while differing from the expressive quality of human-written summaries.
  • Abstractive Summarization: Abstractive summarization generates sentences distinct from the source through structured combination or generative sequence-to-sequence models.Tree-based, graph-based, deep-learning, Transformer, and pre-trained architectures are discussed.
  • Neural and Pre-trained Models: Transformers improve ATS by using attention mechanisms to process long sequences and capture hierarchical or long-range dependencies.Pre-trained models such as BART and Pegasus lower the need to train summarization systems from scratch.
  • Abstractive Summarization: Abstractive models provide more human-like text, flexibility, and compression, but require high-quality data, substantial computation, and longer training.These requirements create a cost-efficiency trade-off during development and deployment.

7. Large Langue Models (LLMs) based Summarization Methods

LLM-based ATS methods extend summarization beyond fixed extractive or abstractive paradigms through prompt engineering, retrieval, fine-tuning, and distillation. This section focuses on prompt engineering, including templates, stepwise reasoning, and agent interactions.

  • Overview: LLMs support summarization, question answering, and logical reasoning, with ATS results reported as comparable to or exceeding human performance.The review reports that LLM summaries can match human-crafted summaries in quality and paraphrasing diversity.
  • Overview: LLM-based ATS methods are categorized into prompt engineering, Retrieval Augmented Generation, fine-tuning, and knowledge distillation.
  • Prompt Engineering: Prompt engineering refines prompts to guide LLMs in generating summaries while reducing the need for extensive training and allowing effective operation with few examples.The reviewed categories include Template Engineering, Chain of Thought, and Agent Interactions.
  • Template Engineering: Template-based prompting uses structured instructions, contextual anchors, and output specifications to guide extraction, condensation, and regeneration.Examples include dual templates for knowledge-graph summarization and perspective-specific templates for healthcare answers.
  • Limitations: Manual prompt design is time-consuming, expertise-dependent, and difficult even for experienced designers, motivating automated template construction.One proposed approach uses entity chains mentioned in target summaries as a basis for automating template design.
  • Chain of Thought: Chain-of-Thought prompting decomposes complex summarization into sequential reasoning steps before producing a coherent summary.These steps can include identifying entities, discerning hierarchical relationships, and synthesizing core arguments.

7.2. Retrieval Augmented Generation for ATS

RAG-based ATS supplements language models with externally retrieved information before generating summaries. The survey describes graph-based and domain-specific variants that target contextual coverage, factual accuracy, and specialized knowledge needs.

  • Graph-Based RAG: Graph RAG constructs entity-and-relationship indexes, partitions them into coherent communities, and retrieves community-level summaries for global context-aware reasoning.
  • RAG Pipeline: RAG-based summarization first retrieves contextually relevant information from external knowledge bases, then integrates it with the input document to generate abstractive summaries.The two-stage process uses retrieval methods such as BM25 or dense passage retrieval followed by generation.
  • Graph-Based RAG: Context-Aware Fine-Grained Graph RAG expands entities through contextual relationships and performs query-level fine-grained summarization to align generation with user intent.
  • Domain Applications: Specialized RAG frameworks address high-volume experimental, clinical, biomedical, legal, and scientific summarization by integrating external domain knowledge.Examples include RAGS4EIC, hybrid clinical extraction and abstraction, medical knowledge-base retrieval, and RAG-RLRC-LaySum.
  • Benefits: RAG mitigates hallucinated or outdated information from fixed training data by dynamically accessing external databases or domain-specific corpora.The survey links this grounding to improved factual accuracy and contextual relevance, particularly in specialized domains.

7.3. Fine-tuning LLMs for ATS

Fine-tuning adapts LLMs to specialized domains and linguistic conventions through internal parameter updates or external adapters. The survey also covers reinforcement learning, LoRA, and distillation-related efficiency trade-offs.

  • Approaches: Fine-tuning adapts LLMs to specialized domains by improving their handling of domain-specific knowledge, linguistic conventions, and task requirements.The reviewed approaches include internal parameter-efficient fine-tuning and external adapter fine-tuning.
  • Internal Fine-Tuning: Internal parameter-efficient fine-tuning freezes most model parameters and updates only a small subset, such as embedding or output layers.
  • Applications: Fine-tuned LLM frameworks have been applied to biomedical literature, accessible lay summaries, and perspective-aware healthcare question-answering.Reported designs include extract-then-summarize pipelines and adapted prompt strategies.
  • Alignment: RLHF uses preference-based reward models encoding conciseness, factual accuracy, and coherence to iteratively optimize summary generation.
  • External Adapters: External adapters learn target-data distributions without modifying the original LLM parameters, supporting few-shot and supervised ATS adaptation.
  • Parameter Efficiency: LoRA adapts pre-trained weights with low-rank matrices, reducing computational and memory costs while maintaining summarization performance.
  • Limitations: Fine-tuning offers domain adaptation and parameter efficiency but faces challenges including overfitting and other limitations in specialized settings.
  • Knowledge Distillation: Knowledge distillation trains a smaller student model under supervision from a larger teacher LLM, enabling more efficient deployment.Distillation can preserve factual consistency and coherence, but effectiveness depends on mitigating inherited flaws and domain-specific fine-tuning.

8. Evaluation Metrics

ATS evaluation uses overlap-based, similarity-based, factual-consistency, and LLM-based metrics to assess summary quality across dimensions such as relevance, factuality, fluency, coherence, and conciseness. The survey also describes newer evaluators that provide fine-grained, human-aligned, or interpretable assessments.

  • Evaluation dimensions: Evaluation should consider relevance, factual consistency, conciseness, semantic coherence, redundancy, and informativeness rather than a single quality dimension.These dimensions are identified as important in prior evaluation work.
  • Overlap-based metrics: Overlap-based metrics compare words or n-grams between candidate and reference summaries using precision, recall, and F-score.ROUGE emphasizes recall, while BLEU emphasizes precision and applies a brevity penalty for short candidates.
  • Overlap-based metrics: Word-overlap metrics do not account for word order or contextual meaning, limiting their assessment of fluency, conciseness, and expressive variation.ROUGE primarily emphasizes recall, and BLEU does not consider grammatical diversity or expressive variations.
  • Similarity-based metrics: Similarity-based metrics use contextual embeddings, generation likelihood, or entailment to assess semantic similarity and factual consistency.Examples include BERTScore, BARTScore, FactCC, and SummaC.
  • LLM-based metrics: Score-based LLM metrics enable quantitative comparison across outputs but depend on predefined criteria and may require high-quality reference summaries.The interpretation of qualities such as fluency and relevance may vary across evaluators.
  • LLM-based metrics: LLM-based evaluators provide continuous scores, human-inspired judgments, pairwise comparisons, or multi-agent debates for more interpretable and human-aligned assessment.Reported methods include ACUEval, Humanlike, DRPE, FENICE, and DEBATE; ACUEval improved balanced accuracy over prior metrics by 3%.

9. Applications based on ATS

ATS supports information access and analysis across news, novels, scientific papers, blogs, dialogues, and medical texts. Applications address domain-specific structures, temporal context, participant variation, citation relationships, and factual accuracy.

  • Cross-domain applications: ATS aims to improve information retrieval and analysis while reducing reading effort in real-world domains.The survey identifies news articles, scientific papers, and other information-rich materials as application areas.
  • News summarization: News summarization is extensively studied because mature datasets support research and industries require timely summaries.Reported approaches include sentence fusion and clustering-based systems for real-time news.
  • Novel summarization: Novel summarization condenses long fictional or literary texts using alignment scores and methods centered on article attributes and dependency relationships.ROUGE and METEOR are used to identify suitable sentences for summary construction.
  • Scientific-paper summarization: Scientific-paper summarization must handle citation relationships, specialized structural parsing, and accurate proper names.CGSum integrates papers and references through citation graphs, while SAPGraph uses a structure-aware heterogeneous graph.
  • Blog summarization: Blog and microblog summarization models participant-linked events and social-temporal context to select significant posts or sub-events.The cited work addresses both global and local temporal context in Twitter data.
  • Dialogue summarization: Dialogue summarization condenses meetings, chats, and emails despite heterogeneous language styles and participant roles.Query-based systems can select conversational discourse using overall content and phrase-specific query information.
  • Medical summarization: Medical summarization can expedite clinical workflows, reduce redundant labor, and enhance clinical communication while requiring factual-accuracy evaluation.One cited framework uses information extraction for automated fact-checking of citations in generated summaries.

10. Future Directions in Large Language Model-Based Summarization

Future research in LLM-based ATS must address domain adaptation, factual reliability, long-document scalability, and interpretability. Proposed directions include specialized training, factuality-aware generation and evaluation, long-context strategies, and transparent prompting.

  • Mitigating Hallucination and Enhancing Trust in LLMs for ATS: Hallucinations can produce fabricated details, misrepresentations, and unsupported claims, compromising the reliability and trustworthiness of ATS outputs.
  • Mitigating Hallucination and Enhancing Trust in LLMs for ATS: Future work should develop factuality-aware evaluation, retrieval- or evidence-grounded generation, and reinforcement learning with trust-oriented reward signals.
  • Scalable Long-Document Summarization with LLMs: Current LLM-based ATS systems struggle with long documents because context windows of roughly 2K to 32K tokens limit document coverage.
  • Scalable Long-Document Summarization with LLMs: Memory-augmented, recurrent, retrieval-augmented, chunk-wise, and hierarchical strategies are proposed to improve long-document scalability and contextual cohesion.
  • Improving Interpretability in LLM-Based ATS: LLM-based ATS remains difficult to interpret because users cannot easily verify the provenance, relevance, or justification of generated information.
  • Improving Interpretability in LLM-Based ATS: Chain-of-Thought prompting and structured prompt engineering are proposed to expose intermediate reasoning or cite relevant input passages.
  • Domain-Specific Summarization with LLMs: General-purpose LLM performance often degrades on domain-specific summarization requiring specialized terminology, complex discourse handling, and high factual precision.
  • Domain-Specific Summarization with LLMs: Domain-adaptive fine-tuning and parameter-efficient adaptation are proposed to improve LLM summarization for specialized corpora while reducing retraining costs.

11. Conclusion

The survey reviews ATS from conventional methods through flexible LLM-based approaches and analyzes how prompting and in-context learning reshape the field. It also identifies hallucination, domain adaptation, efficiency, and explainability as continuing challenges.

  • The survey provides an up-to-date review of extractive, abstractive, and LLM-based ATS methods, emphasizing the latter’s paradigm flexibility.
  • It analyzes how in-context learning, prompt engineering, and few-shot learning have reshaped LLM-based summarization research.
  • The survey proposes an LLM-based retrieval algorithm for efficiently searching and organizing relevant research papers, with potential applications beyond ATS.
  • Future research should improve prompt design, specialized-domain fine-tuning, and evaluation of consistency, factual accuracy, and interpretability.

Appendix B. Author Biographies

The appendix profiles five authors affiliated primarily with Southwestern University of Finance and Economics or Sichuan Agricultural University. Their research interests span text mining, NLP, finance, decision making, and data intelligence.

  • Yang Zhang is an assistant professor at Southwestern University of Finance and Economics, with research interests including text mining and text recommendation.
  • Hanlei Jin is pursuing a Ph.D. in Management Science and Engineering at Southwestern University of Finance and Economics and studies text mining, text generation, and financial intelligence.
  • Dan Meng is a professor at Southwestern University of Finance and Economics whose interests include intelligent finance, intelligent decision making, and uncertainty information processing.
  • Jun Wang is a professor at Southwestern University of Finance and Economics researching NLP, social media, social networks, financial analysis, and business intelligence.
  • Jinghua Tan is an associate professor at Sichuan Agricultural University whose research interests include data mining and financial intelligence.
Loading 2403.02901v3…