Source-linked AI summary
WikiHow: A Large Scale Text Summarization Dataset
Mahnaz Koupaee, William Yang Wang
TL;DR
Summarization needs large, diverse datasets beyond predominantly news-based corpora, especially for higher levels of abstraction. The paper constructs WikiHow from procedural articles and evaluates existing systems, presenting it as a challenging benchmark for abstractive summarization.
Problem
Existing summarization datasets are mostly news-based and limited in scale, stylistic diversity, or abstraction, constraining broader evaluation of summarization systems.
Method
The paper constructs WikiHow by merging procedural article paragraphs and their summarizing outlines, and introduces abstractedness and compression-ratio metrics.
Results
WikiHow contains more than 200,000 long-sequence pairs, and existing systems perform substantially worse on WikiHow than on CNN/Daily Mail, with Lead-3 performing poorly on WikiHow.
Takeaways & Limitations
WikiHow provides a diverse, more abstractive benchmark for evaluating summarization systems and studying challenges beyond news writing.
Abstract
from arXiv · showhide
Sequence-to-sequence models have recently gained the state of the art performance in summarization. However, not too many large-scale high-quality datasets are available and almost all the available ones are mainly news articles with specific writing style. Moreover, abstractive human-style systems involving description of the content at a deeper level require data with higher levels of abstraction. In this paper, we present WikiHow, a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and therefore represent high diversity styles. We evaluate the performance of the existing methods on WikiHow to present its challenges and set some baselines to further improve it.
1 Introduction
Existing summarization datasets are concentrated in news and have limitations in scale, style diversity, abstraction, or task suitability. WikiHow addresses these gaps with a large, diverse dataset, abstraction metrics, and baseline evaluations.
- Large-scale datasets are important for training sequence-to-sequence summarization models, while article length and stylistic diversity add difficulty.
- Most existing datasets use news articles, so systems trained only on them may not generalize well beyond journalistic writing styles.
- WikiHow provides more than 200,000 long-sequence article-summary pairs covering varied topics and writing styles.
- The dataset introduces level of abstractedness and compression ratio metrics to characterize how abstractive the summaries are.
- Existing extractive and abstractive systems are evaluated on WikiHow to establish benchmarks and expose its summarization challenges.
2 Existing Datasets
Prior summarization datasets are largely news-based and differ in size, summary length, and abstraction. Their construction and journalistic conventions shape which summarization settings they support.
- News articles commonly follow an inverted-pyramid style that places important information in opening paragraphs, favoring Lead-3 baselines.
- DUC contains 500 news articles with human-written summaries capped at 75 bytes and multiple summaries per article.
- Gigaword supports short-text summarization through pairs formed from article first lines and headlines, rather than original paired summaries.
- New York Times has mainly been used for extractive summarization, limiting its established role relative to abstractive settings.
- NEWSROOM combines abstractive and extractive strategies but remains a news dataset with average summaries limited to 26.7.
3 WikiHow Dataset
WikiHow articles describe procedural tasks across diverse topics and methods. Their step summaries and detailed descriptions provide article-summary pairs shaped by an inverted relationship between concise outlines and procedural content.
- WikiHow contains procedural articles spanning topics from arts and entertainment to computers and electronics, with single-method or multi-method tasks.
- Each WikiHow step begins with a bold line summarizing that step, followed by its detailed description.
- The dataset constructs summaries by concatenating extracted bold step lines and constructs articles from the remaining detailed descriptions.
- The crawler obtained 142,783 unique articles across 20 categories, with each method treated as a separate article when applicable.
4 WikiHow Properties
WikiHow is a large-scale summarization dataset whose diverse articles, higher abstractedness, and compression ratio distinguish it from news-focused benchmarks.
- More than 230,000 article-summary pairs and longer average articles and summaries make WikiHow larger than DUC and Gigaword.
- WikiHow measures abstractedness using unique summary n-grams absent from the source article.Except for common unigrams, bigrams, and trigrams, WikiHow article-summary pairs share no other n-grams.
- Higher abstractedness requires summarization systems to generate more novel summaries.
- Compression ratio is defined from mean article and summary sentence lengths, with higher values indicating a more difficult task.The paper links higher compression ratios to the need for greater abstraction and semantic capture.
- WikiHow’s higher compression ratio than CNN/Daily Mail indicates a need for higher levels of abstraction.
5 Experiments
The experiments evaluate extractive and abstractive baselines on WikiHow and CNN/Daily Mail, showing that WikiHow is substantially more challenging and does not favor Lead-3.
- The baselines include TextRank, an attention-based sequence-to-sequence model, and a WikiHow-specific Lead-3 baseline.Lead-3 concatenates the first sentence of each paragraph because the first three article sentences cover only a small portion of WikiHow articles.
- The evaluation reports ROUGE-1, ROUGE-2, ROUGE-L, and METEOR using exact-match and stem/paraphrase/synonym criteria.
- 5.2 Results: Baseline systems perform much better on CNN/Daily Mail than on WikiHow, with Lead-3 outperforming other baselines on the news dataset.
- 5.2 Results: About 10 ROUGE points separate CNN/Daily Mail and WikiHow across the baselines.The paper interprets this difference as evidence of additional dataset features and challenges for summarization systems.
6 Conclusion
The paper presents WikiHow as a large-scale dataset of diverse articles and positions it as a resource for evaluating summarization systems.
- WikiHow is a large-scale summarization dataset consisting of diverse articles from the WikiHow knowledge base.
- The dataset’s features create new challenges for summarization systems.
- The authors hope researchers will use WikiHow to evaluate their systems.