Source-linked AI summary
Hierarchical Transformers for Multi-Document Summarization
Yang Liu, Mirella Lapata
TL;DR
Multi-document summarization must handle large document collections despite limited suitable training data. The paper proposes a hierarchical Transformer that learns cross-document relations through attention and can incorporate graph structure, achieving substantial improvements over strong baselines on WikiSum. The authors also identify benefits from learned document ranking and several architectural components.
Problem
Multi-document summarization is limited by scarce high-quality training data and source collections whose size makes encoding all documents impractical.
Method
The model hierarchically encodes selected paragraphs with local and global Transformer layers, using cross-document attention and optionally graph-based structural information.
Results
The architecture brings substantial improvements over several strong baselines on WikiSum, with discourse-graph integration boosting ROUGE-L by 0.16.
Takeaways & Limitations
Hierarchical encoding, learned document selection, and structural information support fluent and informative multi-document summaries.
Takeaways & Limitations
The approach assumes access to large, labeled source-document–summary datasets.
Abstract
from arXiv · showhide
In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial improvements over several strong baselines.
1 Introduction
Multi-document summarization is difficult because training data are scarce and document collections are too large for straightforward end-to-end encoding. The paper addresses these limits with hierarchical cross-document modeling and reports substantial WikiSum improvements over strong baselines.
- Multi-document summarization produces summaries from clusters of thematically related documents.
- Training neural models is constrained by the scarcity of large, high-quality multi-document summarization datasets.DUC and TAC datasets contain only a few hundred examples, while neural methods require more data.
- The large size and number of source documents make encoding all documents into vectors practically infeasible under current hardware memory limits.
- Flat concatenation ignores document hierarchy and cross-document relations such as repetition, additional content, contradiction, and differing perspectives.Cross-document links are relevant to isolating salient information, eliminating redundancy, and producing coherent summaries.
- The proposed model hierarchically encodes documents and uses attention to share information across them while learning latent structural dependencies.Inter-document attention can be replaced with graph matrices based on lexical similarity or discourse relations.
- WikiSum experiments show substantial improvements over several strong baselines, while a document-usefulness ranking module can further boost performance.
2 Related Work
Prior work largely relies on extractive graph-based methods or limited abstractive approaches, while Liu et al. introduced large-scale data and a two-stage system. This paper instead assumes labeled source-summary pairs and combines learned ranking with hierarchical abstractive encoding.
- Previous multi-document summarization methods are predominantly extractive and operate over graph-based sentence or passage representations.They vary in edge weighting, including tf-idf cosine similarity and discourse relations, and in ranking algorithms.
- Abstractive approaches include sentence fusion, whereas neural abstractive systems had achieved promising results mainly for single-document summarization.
- Liu et al. construct large-scale summarization datasets and use a two-stage system that extracts salient information before generating summaries.
- The system ranks numerous or lengthy input paragraphs and summarizes only the L′-best ones with an encoder-decoder model.
- This paper assumes large labeled source-document–summary datasets and contrasts its learned ranker and hierarchical encoder with Liu et al.’s approach.The encoder learns latent cross-document relations and can incorporate information from graph representations.
3 Model Description
The model ranks input paragraphs before abstractive generation, then hierarchically encodes selected paragraphs with local and global Transformer layers. Multi-head pooling and inter-paragraph attention capture within-paragraph context and cross-document dependencies, while explicit graphs can replace a learned attention head.
- The task generates a Wikipedia article’s lead section from its title and a collection of source-document paragraphs.
- Paragraph Ranking: A learning-based ranker scores all paragraphs, orders them by predicted usefulness, and selects the top L′ paragraphs for the abstractive stage.The ranker uses LSTM representations of the title and each paragraph, followed by a linear transformation and sigmoid score.
- Paragraph Encoding: The hierarchical encoder replaces one long selected-paragraph sequence with stacked local and global Transformer layers that capture inter-paragraph relations.Local layers encode contextual information within each paragraph, while global layers exchange information across paragraphs.
- Global Transformer Layer: Multi-head pooling computes weighted paragraph representations by attending to different words and representation subspaces.Each head produces attention scores and a probability distribution over tokens before weighted summation and transformation.
- Global Transformer Layer: Inter-paragraph attention lets each paragraph attend to other paragraphs, producing context vectors that are combined with token representations.Multiple heads allow different attention distributions and views of the input.
- Graph-informed Attention: External graphs can be incorporated by substituting one learned attention head with a graph matrix representing paragraph-edge weights.This provides a mechanism for incorporating relations such as similarity or discourse structure into the hierarchical Transformer.
4 Experimental Setup
The experiments use WikiSum and compare hierarchical Transformer models with extractive and flat Transformer baselines under specified ranking, tokenization, training, and decoding configurations.
- Dataset: WikiSum inputs average 525 paragraphs with 70.1 tokens per paragraph, after crawling 78.9% of the original documents and removing clone paragraphs.Clone paragraphs were removed when their bigram recall against the target summary exceeded 0.8.
- Training and decoding: All abstractive models use shared 32,000-subword SentencePiece tokenization, dropout, label smoothing, Adam optimization with warmup and decay, checkpoint averaging, beam search, and length penalties.Decoding uses beam size 5 and α = 0.4; test results are averaged over the 5 best validation checkpoints.
- Paragraph ranking: The ranking model selects L′ paragraphs using learned scores based on paragraph relatedness to the target summary, contrasting with tf-idf title similarity.The regression model uses ROUGE-2 recall against the target summary as its ground-truth score.
- Comparison systems: The comparison includes Lead and LexRank extractive systems, Flat Transformer variants, T-DMCA, and the proposed Hierarchical Transformer.Lead extracts the first k tokens, LexRank ranks paragraphs using a graph, and Flat Transformer concatenates ranked paragraphs into a truncated token sequence.
- Model configurations: The Hierarchical Transformer uses 7 layers, with 5 local-attention layers and 2 global-attention layers, taking the title and L′ = 24 paragraphs, approximately 1,600 input tokens.This contrasts with flat processing of concatenated token sequences.
5 Results
The experiments evaluate automatic and human judgments of the proposed hierarchical transformer, including ablations, graph variants, and longer test-time inputs. The model generally outperforms baselines, with human participants overwhelmingly preferring it.
- Automatic Evaluation: ROUGE-1, ROUGE-2, and ROUGE-L assess informativeness and fluency in the automatic evaluation.ROUGE-1 and ROUGE-2 measure unigram and bigram overlap, while ROUGE-L measures longest common subsequence.
- Automatic Evaluation: The 1,600-token Hierarchical Transformer outperforms Flat Transformer and T-DMCA evaluated with 3,000 tokens.The comparison is reported in Table 2 across the evaluated systems and input lengths.
- Graph Variants: A discourse graph boosts ROUGE-L by 0.16, while the similarity graph has no obvious influence on results.External graphs are incorporated to enhance the summarization process.
- Automatic Evaluation: Testing the Hierarchical Transformer with 3,000 input tokens improves summarization quality across the board.The authors state that this may improve summaries without increasing training time, whereas the other Transformer models did not show this behavior.
- Ablation Studies: Removing paragraph position, multi-head pooling, or the global transformer layer reduces performance in the ablation studies.The ablations assess the contribution of paragraph position, multi-head pooling, and the global transformer layer.
- Human Evaluation: Participants overwhelmingly prefer the Hierarchical Transformer on informativeness, fluency, and succinctness, with all pairwise comparisons significant at p < 0.01.Human evaluations used Best-Worst Scaling and a question-answering study on summaries produced by the compared systems.
6 Conclusions
The paper frames abstractive multi-document summarization as a machine learning problem and proposes a model that hierarchically encodes documents, learns cross-document relations, and incorporates graph structure. Experiments show fluent and informative summaries that outperform competitive systems by a wide margin.
- 6 Conclusions: The proposed model hierarchically encodes multiple documents, learns latent relations across them, and incorporates structural information from graph representations.The paper also demonstrates the importance of learning-based document selection.
- 6 Conclusions: Experimental results show that the model produces fluent and informative summaries, outperforming competitive systems by a wide margin.The authors identify question answering and related textual inference as future application areas.
A Appendix
Similarity and discourse graphs are added to the hierarchical transformer model to enhance summary quality.
- A Appendix: The similarity and discourse graphs are incorporated into the hierarchical transformer as mechanisms intended to enhance summary quality.The graphs are discussed in Section 3.2.4 and evaluated in Section 5.
A.1 Similarity Graph
The similarity graph represents paragraphs as nodes and weights paragraph connections using tf-idf cosine similarity, retaining edges above a threshold.
- A.1 Similarity Graph: The similarity graph represents each paragraph as a bag of words and uses paragraphs as graph nodes.The graph is based on tf-idf cosine similarity.
- A.1 Similarity Graph: Each paragraph receives a tf-idf vector, and cosine similarity between paragraph pairs becomes the edge weight.Edges with weights lower than 0.2 are removed.
A.2 Discourse Graphs
The paper constructs a simplified Approximate Discourse Graph with paragraph nodes and two edge features: shared entities and explicit discourse markers. Edge weights combine entity co-occurrence with marker-based adjacency signals.
- Graph construction: The simplified Approximate Discourse Graph (ADG) represents paragraphs as nodes and retains two features from the original graph.The construction follows prior work but omits the original ADG’s more complex feature set.
- Graph construction: For each paragraph, named entities are extracted with SpaCy3 using nine specified entity types.The retained types include PERSON, ORG, GPE, LOC, EVENT, and WORK OF ART, among others.
- Edge features: Entity similarity between paragraph pairs is measured by e_ij, the number of exactly matching entities.Each paragraph is represented by an entity set before pairwise overlap is counted.
- Edge features: Discourse-marker edges are identified between adjacent paragraphs in a source webpage using 36 explicit markers.The marker inventory includes terms such as “however,” “moreover,” “similarly,” and “as a result.”
- Edge features: For adjacent paragraphs, m_ii′ equals 1 when one listed discourse marker connects them and 0 otherwise.This binary feature applies only to paragraph pairs adjacent within one source webpage.
- Edge weighting: The final edge weight D_ii′ is a weighted sum of entity co-occurrence e_ii′ and discourse-marker presence m_ii′.Thus, the ADG combines similarity and discourse signals in a single edge weight.