PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-document Summarization via Deep Learning Techniques: A Survey

Congbo Ma, Wei Emma Zhang, Mingyu Guo, Hu Wang, Quan Z. Sheng

arXiv:2011.04843v3cs.CLcs.LG

TL;DR

MDS needs methods that aggregate related documents while handling redundancy, contradiction, long inputs, and limited evaluation support. This survey systematically organizes deep learning MDS models, objectives, metrics, datasets, and network designs, and concludes by identifying open problems and future directions. It reports cross-cutting patterns in objective functions and evaluation practice while documenting scope limitations of current models and datasets.

  • Problem

    MDS must retain critical, coherent, non-redundant, factual content from complex document clusters containing diverse information, while existing surveys largely emphasize traditional or general summarization methods.

  • Method

    The survey proposes a taxonomy and synthesizes deep learning MDS models, network design strategies, objective functions, evaluation metrics, datasets, and future research directions.

  • Results

    Cross-entropy is the most common MDS objective, while reconstruction, redundancy, and max-margin objectives complement it; ROUGE-1, ROUGE-2, and ROUGE-L are the most commonly used metrics.

  • Takeaways & Limitations

    The taxonomy and synthesis provide a comprehensive understanding of current deep learning MDS approaches and clarify open problems and promising future extensions.

  • Takeaways & Limitations

    Current deep learning MDS models require more training data, data-labeling effort, and computing resources, while CNNs remain challenged by non-local relationships in lengthy inputs.

Abstract

from arXiv · show

Multi-document summarization (MDS) is an effective tool for information aggregation that generates an informative and concise summary from a cluster of topic-related documents. Our survey, the first of its kind, systematically overviews the recent deep learning based MDS models. We propose a novel taxonomy to summarize the design strategies of neural networks and conduct a comprehensive summary of the state-of-the-art. We highlight the differences between various objective functions that are rarely discussed in the existing literature. Finally, we propose several future directions pertaining to this new and exciting field.

1 INTRODUCTION

MDS addresses the need to efficiently condense abundant, related text while resolving redundancy, contradiction, and diverse perspectives across documents. This survey organizes deep learning approaches, objectives, datasets, evaluation, and future directions.

  • MDS generates comprehensive summaries from documents covering different perspectives, but must resolve diverse and redundant information.
  • Long and complex inputs make it difficult to retain critical content while producing coherent, non-redundant, factual, and grammatically readable summaries.
  • MDS supports applications including news, scientific publications, emails, product reviews, medical documents, and Wikipedia generation.
  • The survey systematically reviews deep learning MDS publications and proposes a taxonomy covering models, objectives, datasets, metrics, and network design strategies.
  • The survey also discusses open issues and identifies future research directions for deep learning based MDS.

2 FROM SINGLE TO MULTI-DOCUMENT SUMMARIZATION

MDS extends summarization from one document to related document clusters, requiring models to capture cross-document relations and manage conflict, duplication, and complementarity. The section contrasts shared summarization principles with MDS-specific input and concatenation challenges.

  • Similarities between SDS and MDS: MDS aims to generate a concise, informative summary from a cluster of topic-related documents, whereas SDS summarizes one document.
  • Similarities between SDS and MDS: Summarization methods include extractive, abstractive, and hybrid construction types, with hybrid models combining extractive and abstractive advantages.
  • Differences between SDS and MDS: MDS differs from SDS through more diverse inputs, insufficient cross-document relation modeling, redundancy and contradiction, larger search spaces, and limited training data or evaluation metrics.
  • Differences between SDS and MDS: Flat concatenation processes all documents as one sequence, while hierarchical concatenation preserves cross-document relations through document- or word/sentence-level processing.
  • Differences between SDS and MDS: Conflict, duplication, and complementarity require MDS models to identify and handle complex information beyond directly applying SDS models.

3 DEEP LEARNING BASED MULTI-DOCUMENT SUMMARIZATION METHODS

Deep learning MDS systems use varied architecture strategies to generate or fuse semantic- and syntactic-rich representations. The survey organizes these strategies, including naive, ensemble, fusion, graph, and encoder-decoder designs.

  • Architecture Design Strategies: The survey groups MDS architectures into nine types based on how they generate or fuse semantic-rich and syntactic-rich representations.
  • Architecture Design Strategies: Architecture strategies can serve as basic structures or be stacked to obtain more diverse MDS designs.
  • Architecture Design Strategies: Naive networks process concatenated documents with deep neural models using word-, sentence-, or document-level representations.
  • Architecture Design Strategies: Ensemble networks use multiple network paths and fuse their representations to enhance model expression capability.
  • Architecture Design Strategies: Fusion networks combine neural representations with hand-crafted features, while graph neural networks model word-, sentence-, or document-level relations across documents.

3.2 Recurrent Neural Networks based Models

RNN-based MDS models represent sequential and hierarchical document information, with variants for extractive, abstractive, unsupervised, and hybrid summarization. Their sequential modeling strength is accompanied by training and long-sequence limitations.

  • RNN foundations: RNNs model sequential relations and syntactic or semantic information through hidden states connected across input steps.Each neuron receives the current embedding and the previous hidden state, but vanilla RNNs can suffer gradient explosion or vanishing issues.
  • RNN foundations: R2N2 transforms sentence ranking into hierarchical regression using manually extracted word- and sentence-level features.The framework measures sentence and constituent importance within a parsing tree.
  • Hybrid and unsupervised variants: Hybrid RNN frameworks combine Bi-LSTM representations, multi-source fusion, attention, and copying for opinion summarization.A two-stage design first learns word- and document-level representations, then generates the summary with an LSTM decoder.
  • Hybrid and unsupervised variants: Unsupervised RNN frameworks use salient information vectors, cascaded attention, reconstruction, and sparsity constraints when paired MDS datasets are scarce.The sparsity constraint penalizes trivial information in reconstructed output vectors.
  • Architectural trade-offs: CNN-RNN comparisons expose a trade-off between RNN sequence modeling and CNN parallelism, while CNNs remain challenged by non-local relations in lengthy inputs.RNNs are difficult to parallelize and may lose earlier knowledge; CNNs train more efficiently but fixed receptive fields hinder long-distance modeling.

3.4 Graph Neural Networks Based Models

Graph neural network models represent relationships among sentences, documents, and other linguistic units to support MDS. The surveyed designs range from sentence graphs to heterogeneous word-sentence-document graphs.

  • Graph representations: GNNs suit NLP because graph structures model semantic and syntactic relations that sequential representations may not capture directly.GCNs and GANs are the most common GNN variants described for MDS.
  • Sentence-level graphs: A GCN extractive model treats sentences as nodes and sentence relations as edges, then aggregates sentence and document features for salience estimation.Relations can be built with cosine similarity, discourse graphs, or a personalized discourse graph.
  • Sentence-level graphs: SemSentSum constructs a sentence semantic relation graph with external universal embeddings and removes low-weight edges to address sparse graphs.Edges below a specified similarity threshold are removed.
  • Scientific-paper summarization: ScisummNet combines abstracts, citing-paper text, citation counts, cosine-similarity graphs, and GCNs to summarize scientific papers.Citation count is incorporated as an authority feature.
  • Heterogeneous graphs: HeterDoc-SUM Graph uses word, sentence, and document nodes, with TF-IDF-weighted edges and bidirectional graph updates to aggregate cross-level semantic knowledge.Word nodes bridge sentence and document nodes, enabling document-document, sentence-sentence, and sentence-document relations.

3.5 Pointer-generator Networks Based Models

Pointer-generator and related attention architectures address factual errors, redundancy, long inputs, and computational constraints in MDS. The surveyed approaches combine copying or redundancy control with increasingly parallelizable Transformer designs.

  • Pointer-generator designs: Pointer-generator networks combine sequence-to-sequence and pointer networks into a unified distribution over generated vocabulary and source-text tokens.They were proposed to address factual errors and high redundancy in summarization.
  • Redundancy control: MMR selects salient sentences by jointly considering importance and redundancy until a summary-length threshold is reached.The redundancy score limits overlap with the existing summary.
  • Transformer motivation: RNN models face limited parallelization and difficulty retaining information across long sequences, motivating Transformer-based alternatives.RNN states depend on previous steps, whereas long inputs can cause earlier knowledge to fade.
  • Transformer architectures: Transformer architectures use self-attention for parallelization and relative long-range dependencies, commonly following encoder-decoder designs in MDS.The survey describes flat and hierarchical Transformer variants.
  • Long-document scaling: Local and global attention, multi-layer Transformers, and block-level information sharing are used to scale long-document encoding for MDS.These approaches allow pretrained language models to process smaller text blocks while sharing information across successive layers.

3.7 Deep Hybrid Models

Deep hybrid MDS models combine multiple neural architectures or formulate specialized variants to improve representation, selection, and generation. The section also situates these designs across query-oriented, dialogue, and reinforcement-learning-related tasks.

  • Hybrid architectures: Hybrid models integrate multiple neural networks to produce more powerful and expressive MDS representations.The survey reviews CNN, LSTM, capsule, and other combinations.
  • Task variants: Hybrid MDS techniques are applied to task variants including query-oriented summarization, which selects information that answers a query in logical order.Query-oriented MDS summarizes a document set according to the information most relevant to the query.
  • Method taxonomy: The surveyed method taxonomy distinguishes extractive, abstractive, and hybrid approaches, along with flat and hierarchical concatenation.The table abbreviations are Ext, Abs, Hyd, FC, and HC.
  • Model trade-offs: PG-MMR is characterized by low redundancy but difficult training, whereas hierarchical Transformer models offer parallel computing and cross-document and in-document relation modeling with time and position-encoding costs.These trade-offs are summarized in the model comparison table.
  • Task variants: Deep MDS variants include graph, long-document, hybrid, and dialogue-oriented models, while real-time summarization can be formulated as sequential decision making.The real-time formulation uses an LSTM and three fully connected layers to maximize long-term rewards.

3.9 Discussion

Deep learning-based MDS models increasingly use Transformers, but they require more data, labeling, and computing resources and lack linguistic knowledge that can improve representations and summary generation.

  • Transformer-based models have been most commonly used in recent MDS because they overcome CNN receptive-field limits and RNN parallelization limits.
  • Deep learning-based MDS models require more training data, data labeling, and computing resources than non-deep-learning methods.
  • Deep learning-based methods lack linguistic knowledge that could assist representation learning and guide informative summaries.

4 OBJECTIVE FUNCTIONS

MDS models use diverse objectives to guide summary generation, representation learning, redundancy reduction, and auxiliary supervision. Cross-entropy is most common, while complementary objectives address its limited supervision and support optimization from multiple perspectives.

  • Reconstructive objectives train representation learners by reconstructing input vectors in an unsupervised manner.Encoder-decoder reconstruction can constrain generated text toward the natural-language domain, and reconstruction-cycle losses connect reviews and summaries.
  • Redundancy objectives minimize overlap among semantic units, encouraging information coverage and reducing repetitive phrases or content.Similarity can be measured across phrases, sentences, topics, or documents, and redundancy loss is often combined with other objectives.
  • Max-margin objectives separate representations by a predefined margin and can rank sentences according to salience.They constrain function outputs for input pairs whose desired ordering or distinction is specified by the task.
  • Multi-task objectives combine the MDS loss with auxiliary-task losses because MDS supervision may be insufficient for representation learning.Auxiliary tasks provide additional constraints and can support better generalization and optimization from multiple angles.
  • Cross-entropy measures differences between generated-summary and golden-summary distributions and is the most commonly adopted MDS objective.It treats golden summaries as strong supervision signals.
  • Using multiple objectives can complement cross-entropy through unsupervised reconstruction, redundancy constraints, max-margin learning, and auxiliary supervision.This combination can enable optimization using input documents when manual annotation is scarce.

5 EVALUATION METRICS

The survey groups MDS evaluation into ROUGE-based and alternative metrics, noting that ROUGE remains dominant while other measures provide complementary lexical, semantic, linguistic, and content-oriented views.

  • ROUGE: ROUGE compares generated summaries with human-written references and includes variants for different forms of candidate-summary matching.ROUGE-N mainly measures n-gram recall, while ROUGE-L measures sequence similarity through the longest common subsequence.
  • ROUGE: ROUGE-L measures sentence-level sequence similarity using an automatically selected longest common subsequence rather than a manually specified n-gram.Its balance factor controls the relative contribution of LCS recall and precision.
  • ROUGE: Jackknifing averages scores across sets of M-1 reference summaries when multiple golden summaries are available.The procedure applies to ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S.
  • Other Evaluation Metrics: Alternative metrics include precision-oriented BLEU, language-quality perplexity, manually annotated Pyramid scores, and human-rated Responsiveness.These metrics assess different properties, including vocabulary overlap, grammatical quality, content coverage, and linguistic quality.
  • Discussion: ROUGE-1, ROUGE-2, and ROUGE-L are the most commonly used MDS metrics, whereas other indicators currently serve mainly as assistance.The survey also notes potential adaptation of information-retrieval measures such as efficiency, effectiveness, and coverage.

6 DATASETS

The survey reviews MDS datasets across news, scientific, Wikipedia, review, and variant-task settings, emphasizing that large-scale, diverse resources remain scarce and that common news datasets contain structural biases.

  • Dataset landscape: Large-scale MDS datasets covering general scenarios and downstream tasks are relatively scarce compared with SDS datasets.The survey investigates 10 representative datasets commonly used for MDS and related tasks.
  • News datasets: DUC and TAC provide small news-domain datasets with hundreds of documents and human-annotated summaries, limiting their suitability for training large deep neural models.Their news structure also tends to place information-rich content in the first sentence.
  • Dataset landscape: The survey covers datasets for news, scientific publications, Wikipedia, customer reviews, and query-oriented, dialogue, and real-time summarization variants.Examples include Multi-News, WikiSum, Yelp, Scisumm, WCEP, Multi-XScience, Debatepedia, AQUAMUSE, QBSUM, and several dialogue datasets.
  • Discussion: News datasets constitute 60% of the surveyed collection, while scientific papers and Wikipedia each constitute 10%.Recent WikiSum and Multi-News datasets address training requirements through larger-scale data.
  • Dataset limitations: News-document structure can introduce positional and extractive biases during training, while Multi-XScience contains fewer such biases than WikiSum and Multi-News.The survey links these biases to highly compressed information near the beginnings of news articles.

7 FUTURE RESEARCH DIRECTIONS AND OPEN ISSUES

Future MDS research should address cross-document relations, data and metric limitations, long-sequence processing, interpretability and robustness, reinforcement learning, and multimodal inputs. The survey identifies these as open issues for advancing MDS across applications.

  • Cross-document relations: MDS models often concatenate documents into flat sequences, overlooking redundant, complementary, or contradictory cross-document relations.Capturing these relations can help extract salient information, improve coherence, and reduce redundancy.
  • Datasets: DUC and TAC contain relatively few samples for training DNNs, while newer datasets still need greater diversity and fewer positional or extractive biases.The survey also calls for datasets covering medical records, dialogue, email, code, and software projects.
  • Evaluation metrics: MDS lacks evaluation metrics designed specifically for relations among input documents, redundancy reduction, and contradiction handling.ROUGE measures vocabulary-level distance and an improved score does not necessarily indicate higher summary quality.
  • Reinforcement learning: Adapting reinforcement learning to MDS remains challenging because the task has large state and action spaces, redundancy, and contradiction.The survey suggests model-based approaches as a possible route toward more data-efficient reinforcement learning for MDS.
  • Pretrained language models: Long-sequence MDS motivates exploration of pretrained language models that avoid the quadratic memory growth of self-attention.The survey names Longformer, REFORMER, and Big Bird as examples of relevant architectures.
  • Interpretability and robustness: Explainability and adversarial testing are proposed research directions for determining whether MDS models generate condensed, coherent summaries without bias and remain robust.The survey frames deep models as difficult to inspect because of their high non-linearity.
  • Multimodality: Multimodal MDS remains largely under-explored despite possible text-plus-image, text-plus-video, and text-plus-audio applications.Existing multimodal summarization has primarily used non-deep-learning techniques.

8 CONCLUSION

The survey reviews deep learning-based MDS research through a taxonomy of network design strategies, objective functions, evaluation metrics, and datasets. It synthesizes notable advances and identifies open problems and future extensions.

  • Scope: The survey presents a comprehensive review of deep learning-based multi-document summarization research.It describes the review as the first comprehensive review of notable works to date.
  • Contributions: Its taxonomy organizes existing publications and derives network design strategies from state-of-the-art methods.The survey also covers multi-document objective functions, evaluation metrics, and datasets.
  • Future directions: The article discusses pressing open problems and promising future extensions in MDS research.The stated aim is to clarify notable advances and inform future studies.
Loading 2011.04843v3…