Source-linked AI summary
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, Rifat Shahriyar
TL;DR
Abstractive summarization has lacked large datasets for many languages, limiting multilingual and low-resource research. XL-Sum introduces a BBC-derived dataset and curation tool covering 1 million pairs in 44 languages, then evaluates multilingual and individual low-resource training. It reports competitive performance, including ROUGE-2 above 11 across 10 benchmarked languages, while noting evaluation limitations for some low-resource languages.
Problem
Multilingual abstractive summarization datasets remain limited in language coverage and training samples, leaving research centered largely on English and high-resource languages.
Method
XL-Sum collects 1 million professionally annotated BBC article-summary pairs in 44 languages, releases a curation tool, and fine-tunes mT5 for multilingual and individual low-resource summarization.
Results
Multilingual training achieves higher than 11 ROUGE-2 on all 10 benchmarked languages, with some exceeding 15, while individual low-resource training also provides competitive results.
Takeaways & Limitations
XL-Sum provides publicly available abstractive summarization data and benchmarks for many languages, supporting research beyond English, especially for low- and mid-resource languages.
Takeaways & Limitations
Low-resource results are affected by limited training samples and unreliable or aggressive language-specific stemming and segmentation for ROUGE evaluation.
Abstract
from arXiv · showhide
Contemporary works on abstractive text summarization have focused primarily on high-resource languages like English, mostly due to the limited availability of datasets for low/mid-resource ones. In this work, we present XL-Sum, a comprehensive and diverse dataset comprising 1 million professionally annotated article-summary pairs from BBC, extracted using a set of carefully designed heuristics. The dataset covers 44 languages ranging from low to high-resource, for many of which no public dataset is currently available. XL-Sum is highly abstractive, concise, and of high quality, as indicated by human and intrinsic evaluation. We fine-tune mT5, a state-of-the-art pretrained multilingual model, with XL-Sum and experiment on multilingual and low-resource summarization tasks. XL-Sum induces competitive results compared to the ones obtained using similar monolingual datasets: we show higher than 11 ROUGE-2 scores on 10 languages we benchmark on, with some of them exceeding 15, as obtained by multilingual training. Additionally, training on low-resource languages individually also provides competitive performance. To the best of our knowledge, XL-Sum is the largest abstractive summarization dataset in terms of the number of samples collected from a single source and the number of languages covered. We are releasing our dataset and models to encourage future research on multilingual abstractive summarization. The resources can be found at \url{https://github.com/csebuetnlp/xl-sum}.
1 Introduction
XL-Sum addresses limited multilingual abstractive summarization data by releasing a 1-million-pair dataset spanning 44 languages and reporting strong multilingual and low-resource results.
- Motivation: Limited multilingual datasets have constrained abstractive summarization research, which has centered largely on English and other high-resource languages.Existing multilingual efforts were limited by language coverage, training-sample counts, or both.
- Results: Multilingual summarization achieved higher than 11 ROUGE-2 on all 10 benchmarked languages, with many exceeding 15.The reported English state-of-the-art reference is 16.58 ROUGE-2 on XSum.
- Results: Individual training on low-resource languages also produced competitive summarization results.This indicates the dataset can support low-resource languages separately, not only through multilingual training.
- Dataset and contributions: 1 million article-summary pairs across 44 languages comprise XL-Sum, with many languages receiving their first publicly available abstractive summarization dataset.The dataset uses professionally annotated articles from BBC.
- Dataset and contributions: A data curation tool automatically crawls BBC pages and extracts article-summary pairs, allowing the dataset to grow over time.The authors release the curation tool alongside the dataset and model checkpoints.
- Implications: The released resources are intended to encourage abstractive summarization research beyond English, especially for low- and mid-resource languages.The paper connects this scope to traditionally underserved language communities.
2 The XL-Sum Dataset
XL-Sum collects multilingual news articles from BBC and extracts summaries using consistent editorial structure and filtering heuristics across 44 language variants.
- 2.1 Content Source: More than two-thirds of XL-Sum's languages are low-resource under the threshold of fewer than 15,000 training samples.Here, each sample is an article-summary pair.
- 2.1 Content Source: BBC publishes news in 43 languages, while Serbian Cyrillic and Latin are treated separately, yielding 44 covered languages.The dataset spans low-resource languages such as Bengali and Swahili and higher-resource languages including English and Russian.
- 2.1 Content Source: A custom crawler recursively visits article links across BBC sites and discards pages without textual content.The crawler compensates for BBC's lack of an archive or RSS feed by exploiting similar site structures.
- 2.2 Summary Extraction: BBC article summaries are typically bold paragraphs of one or two sentences near the beginning, written to convey the article's main story.The authors distinguish these summaries from headlines, which primarily attract viewers' attention.
- 2.2 Summary Extraction: The extraction heuristics require the summary within the first two paragraphs, some bold text, and at least 95% bold or hyperlinked content.These conditions identify the editorial summary paragraph rather than arbitrary article text.
- 2.2 Summary Extraction: All text except the summary and headline enters the input, and the input must be at least twice as long as the summary.Samples failing any heuristic are discarded.
3 Human Evaluation of XL-Sum
Human evaluation found that XL-Sum summaries generally conveyed the articles’ main ideas without conflicting information, although many contained details not directly inferable from the articles.
- Evaluation setup: Professional annotators evaluated random subsets of XL-Sum for the top 10 languages by worldwide speaker count.The evaluators had undergraduate degrees and native or bilingual proficiency in their assigned languages.
- Agreement: Inter-annotator agreement was high, with most Cohen’s kappa scores between 0.7 and 0.9.The evaluation treated Property A as present when both evaluators agreed, while Properties B and C required agreement from at least one evaluator.
- Summary quality: Most languages had summary-quality rates in the upper nineties for Property A, with somewhat lower rates for Spanish and Portuguese.Property A assessed whether the summary conveyed the article’s main idea.
- Evaluation caveat: Negative summaries were mostly extracted from opinion pieces and blog posts whose bold paragraphs did not convey the articles’ main stories.This identifies a source-content boundary affecting some extracted summaries.
- Factual consistency: Almost none of the summaries contained conflicting information under Property B.Property B assessed consistency between each summary and its article.
- Additional information: About one-third of summaries contained information that was not directly inferrable from the article under Property C.More than 75% of such additions involved missing first names, designations, or acronym elaborations.
4 Intrinsic Evaluation of XL-Sum
Intrinsic evaluation indicates that XL-Sum is highly abstractive, concise, and minimally redundant across languages, with quality comparable to XSum and stronger metrics than CNN/DM.
- Metrics: The evaluation reports novel n-gram, abstractivity, compression, and redundancy metrics for XL-Sum alongside CNN/DM and XSum.For redundancy, the authors report values for n = 1 and n = 2.
- Dataset comparison: XL-Sum and XSum show comparable intrinsic quality, while CNN/DM trails them on most reported metrics.The comparison covers novel n-grams, abstractivity, redundancy, and compression.
- Abstractiveness: About one-third of XL-Sum summary tokens and more than 75% of bigrams are novel, while abstractivity exceeds 65% for most languages.Novel n-gram ratios measure summary n-grams absent from the input article; abstractivity is computed from article-summary fragments.
- Conciseness: XL-Sum summaries are less than one-tenth the length of their input articles for most languages.Compression uses article and summary token counts, with |A| denoting article length and |S| summary length.
- Redundancy: Redundancy is below 10% for the majority of XL-Sum languages.The proposed metric calculates repetitive n-grams because existing redundancy measures target multi-sentence summaries, unlike most XL-Sum examples.
5 Experiments and Benchmarks
The experiments evaluate XL-Sum with mT5 in multilingual and low-resource settings, showing strong multilingual results and competitive performance from individually trained low-resource models.
- Experimental Setup: XL-Sum experiments evaluate mT5 in multilingual and low-resource summarization settings.The multilingual setup trains one model across languages, while the low-resource setup trains separate models for five languages.
- Multilingual Summarization: Higher than 11 ROUGE-2 scores were achieved on all 10 benchmarked languages in multilingual training.The benchmark includes English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Portuguese, and Indonesian.
- Multilingual Summarization: XL-Sum provides the first reported abstractive summarization benchmark for several languages, including Bengali.The multilingual model also produced competitive results for Bengali despite its low-resource status.
- Multilingual Summarization: 15.18 R2-score was achieved by mT5-base on English, compared with 16.58 from PEGASUSBASE on the similar XSum dataset.The paper describes this English performance as comparable to state-of-the-art summarization results.
- Low-Resource Summarization: The multilingual model outperformed every individually trained low-resource model, with R-2 differences no greater than 2.The individually trained languages were Amharic, Azerbaijani, Bengali, Japanese, and Swahili.
- Low-Resource Summarization: Approximately 4k Amharic and 6k Azerbaijani training samples coincided with comparatively low scores, while Japanese scores were affected by aggressive segmentation.The authors also note that unavailable reliable stemming may have hurt Amharic and Azerbaijani ROUGE scores.
6 Related Works
Prior abstractive summarization research developed neural, pointer-generator, convolutional, and pretrained seq2seq approaches, but remained concentrated on English because multilingual datasets were limited.
- Neural Abstractive Summarization: Neural abstractive summarization began with recurrent attentional seq2seq models and later incorporated copying and convolutional architectures.Pointer-Generator networks copy input words while also generating new decoder text.
- Pretrained Models: Pretrained language models were applied by initializing encoders, decoders, or complete seq2seq models before fine-tuning for abstractive summarization.The cited approaches include BERT-initialized models and fully pretrained seq2seq models.
- Multilingual Resources: Multilingual summarization datasets remained limited in size or language coverage compared with English resources.MultiLing 2015 had 10k total training samples, while two newer datasets covered fewer than 10 languages.
7 Conclusion and Future Works
The paper concludes that XL-Sum is a large, high-quality multilingual dataset supporting abstractive summarization across 44 languages, including low-resource and compute-efficient settings.
- Conclusion: XL-Sum contains 1 million high-quality samples across 44 languages collected from BBC, with first public datasets and benchmarks for many languages.The dataset is described as highly abstractive, concise, and faithful based on human and intrinsic evaluations.
- Conclusion: Multilingual training can improve summarization, likely through positive transfer between morphologically similar sister languages.The conclusion also reports usefulness in low-resource and compute-efficient settings.
- Future Work: Future work will investigate XL-Sum for other summarization tasks, including cross-lingual summarization.The authors hope the dataset will support research aimed at fairer access to technology for under-served low-resource-language communities.