Source-linked AI summary
Generating Wikipedia by Summarizing Long Sequences
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer
TL;DR
Generating Wikipedia is posed as multi-document summarization, using extractive selection followed by neural abstractive generation. The paper introduces a decoder-only architecture for very long sequences and reports coherent, informative Wikipedia generation from many reference documents. The approach also exposes the importance of improving extraction and extending long-context decoder-only models.
Problem
Abstractive neural methods have limited application to multi-document summarization, where large labeled datasets and very long inputs make end-to-end generation difficult.
Method
The paper first uses extractive summarization to select salient information, then conditions a decoder-only Transformer language model on that extraction to generate Wikipedia text.
Results
The decoder-only model significantly outperforms traditional encoder-decoder architectures on long sequences and generates coherent, informative Wikipedia articles conditioned on many reference documents.
Takeaways & Limitations
Wikipedia generation can be approached as multi-document summarization, with long-context decoder-only modeling enabling article-scale abstractive generation.
Takeaways & Limitations
Future gains may depend on improving the extractive stage and extending decoder-only architectures to learn from larger L while maintaining sufficient model capacity.
Abstract
from arXiv · showhide
We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.
1 INTRODUCTION
The paper frames English Wikipedia generation as supervised multi-document summarization and introduces a decoder-only model to generate Wikipedia text from reference documents.
- Prior neural abstractive summarization trained end-to-end on article-summary pairs, requiring substantial parallel data for fluent generation.
- Multi-document summarization distills a summary from a collection of related documents, but abstractive neural methods have seen limited application because labeled datasets are scarce.
- The proposed task uses a Wikipedia topic and non-Wikipedia reference documents as input, with the corresponding Wikipedia article as the target.
- The paper modifies the Transformer into a decoder-only architecture and reports better performance on longer inputs than recurrent and encoder-decoder Transformer models.
- The modeling improvements support generation of entire Wikipedia articles, beyond the first-section lead-generation task.
2 RELATED WORK
Prior abstractive summarization addressed shorter or more constrained news settings, whereas this work studies larger, multi-sentence Wikipedia summaries from heterogeneous documents.
- Headline generation uses only an article’s first sentence to predict a headline, making it closer to sentence paraphrasing than full summarization.
- The Daily Mail and CNN task draws highlight information from multiple article locations and has 310k parallel examples versus 3.8M for Gigaword.
- Wikipedia leads contain multiple sentences or paragraphs in a relatively uniform style, while source documents may have arbitrary styles and origins.
- 59.2 ROUGE-1 recall for Wikipedia is lower than 76.1 and 78.7 for other summarization datasets, indicating weaker suitability for purely extractive methods.
- Earlier Wikipedia generation used extractive methods and learned templates for only two article categories, whereas this work generates abstractively across article types.
- Transformers enable greater within-example parallelization than recurrent models but incur quadratic complexity in input sequence length.
3 ENGLISH WIKIPEDIA AS A MULTI-DOCUMENT SUMMARIZATION DATASET
The WikiSum dataset treats Wikipedia articles as summaries of reference documents, combining cited sources with filtered web-search results and providing a very large corpus.
- Wikipedia articles are modeled as topic-titled summaries, with source material approximated using selected subsets of web and book-like documents.
- Cited sources provide extracted text from crawlable reference documents, while web-search results expand coverage using section-title queries after removing Wikipedia clones.
- Many articles have few citations, motivating web-search supplementation, while available citations tend to be higher quality.
- The full WikiSum dataset contains orders-of-magnitude more words than previous summarization datasets.
- The corpus-comparison experiments restrict examples to articles with at least one crawlable citation and split them roughly 80/10/10 into train, development, and test sets.
4 METHODS AND MODELS
The method uses extractive summarization to select salient source paragraphs before an abstractive model generates Wikipedia lead text. A decoder-only Transformer and modified self-attention enable processing substantially longer input sequences.
- 4.1 Extractive Stage: Extractive summarization first ranks source paragraphs and selects the first L tokens as input to the abstractive stage.The system evaluates identity, tf-idf, TextRank, SumBasic, and a cheating extractor.
- Input Representation: The source input concatenates paragraphs in ranked order, prioritizing the most relevant paragraphs, and prefixes the sequence with the article title.Sub-word tokenization uses a 32,000-wordpiece vocabulary; input length L varies across experiments.
- Abstractive Models: The abstractive model treats article generation as sequence transduction from inputs up to L = 11000 tokens to outputs typically shorter than 500 tokens.Wikipedia lead text uses the same vocabulary and tokenization without output truncation.
- Transformer Decoder: The decoder-only Transformer drops the encoder, concatenates input and output with a separator token, and trains as a standard language model.This modification almost halves model parameters for a given hyper-parameter set and converts each example into one combined sentence.
- Long-Sequence Attention: Memory-compressed and local attention reduce self-attention memory costs for long sequences by compressing keys and values or restricting attention to blocks.Local attention uses blocks of 256 tokens, while memory-compressed attention uses convolution kernels of size 3 with stride 3.
- Long-Sequence Attention: The final T-DMCA architecture alternates local and memory-compressed layers in a five-layer LMLML network, enabling sequences 3x longer than the T-D model in practice.Causal masking prevents queries from attending to future keys and values.
5 EXPERIMENTS
Experiments show that smart extraction and long-context decoder-only architectures are central to generating higher-quality Wikipedia leads. The best abstractive models improve both automatic metrics and human-rated linguistic quality, while reference-conditioned models can produce coherent, factually informative text.
- Evaluation: Perplexity optimization correlated with increased ROUGE and human judgment, although distinguishing models became harder at the highest performance levels.The authors attribute this partly to Wikipedia's relatively uniform style.
- Results: The best abstractive model more than doubled extractive-only ROUGE-L F1 and substantially improved perceived linguistic quality.T-DMCA was statistically significantly better than the compared models on all evaluated linguistic dimensions except non-redundancy, where tf-idf performed similarly.
- Extractive method: Smart extraction is critical: tf-idf substantially outperformed doing nothing or identity extraction, while a cheating extractor indicates further extraction gains remain possible.The experiments fixed tf-idf for subsequent model comparisons and identify supervised relevance prediction as future work.
- Input corpus: The combined citation-and-search corpus performed best, with significant and complementary contributions from citations and search results.Subsequent experiments therefore report only combined-corpus results.
- Model architecture and input length: T-DMCA trained up to L = 11000 and continued improving, while a 128-expert MoE reduced log-perplexity from 2.05 to 1.93 at L = 11000.The best attempted model used 256 experts at L = 7500 and achieved perplexity 1.90; 256 experts at L = 11000 exceeded available memory.
- Qualitative discussion: As perplexity decreased, outputs improved in fluency, factual accuracy, and narrative complexity; T-DMCA produced succinct text containing key facts.Full-article samples also showed plausible section organization, global coherence, and factual information from reference documents, though full-article generation was not a primary focus.
6 CONCLUSION
The paper frames Wikipedia generation as multi-document summarization using a two-stage extractive-abstractive framework. Its decoder-only model handles very long sequences and generates coherent, informative Wikipedia articles.
- Wikipedia generation is approached as multi-document summarization with a large, parallel dataset and a two-stage extractive-abstractive framework.
- The decoder-only sequence-transduction model handles very long input-output examples and significantly outperforms traditional encoder-decoder architectures on long sequences.
- The model can condition on many reference documents while generating coherent and informative Wikipedia articles.
- The extractive stage has a significant effect on final performance, making improved extraction a promising direction for further research.
7 PUBLIC RELEASE OF DATASET AND CODE
The authors plan to release the experiment URLs and abstractive modeling code extensions to support further research on large-scale summarization.
- The released experiment resources include Wikipedia URLs and reference-document URLs available through the freely downloadable CommonCrawl dataset.
- The authors will release extensions to the open-source tensor2tensor library used to train the abstractive models.
A.1 EXAMPLES OF FULL WIKIPEDIA GENERATED SAMPLES
The appendix illustrates full-article generation by T-DMCA models under reference-conditioned and title-only conditions. These examples show the models' sampling setups rather than quantitative comparisons.
- A T-DMCA model generates an entire Wikipedia article conditioned on 8192 reference-document tokens.
- Three T-DMCA samples generate entire Wikipedia articles when conditioned only on the article title.Samples 1 and 3 are truncated because of space constraints.
A.2.1 WIKIPEDIA CLONE DETECTION
This appendix covers reference-document clone detection and human evaluation procedures, alongside an example of extractive input produced with tf-idf.
- Wikipedia clone detection: Clone detection computes each reference document's maximum unigram recall against sections of the corresponding Wikipedia article.
- Wikipedia clone detection: A reference document is detected as a clone when its maximum section-level unigram recall exceeds 0.5.
- Human evaluation: Linguistic quality ratings score grammaticality, non-redundancy, referential clarity, focus, and structure and coherence on a 1-to-5 scale.Each model uses 25 examples, with scores averaged across 3 raters.
- Human evaluation: Pairwise human evaluation presents model outputs side by side and records the ratio of ratings preferring one model over the other.The side shown for each model is randomized per example and rater.
- Extractive input: Figure 10 shows an extractive-output/abstractive-input example produced using tf-idf extraction.