PaperlayerView exact revision on arXiv

Source-linked AI summary

MLSUM: The Multilingual Summarization Corpus

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano

arXiv:2004.14900v1cs.CL

TL;DR

Summarization research has lacked large-scale multilingual training data, limiting direct study of language-specific performance. MLSUM introduces a 1.5M-plus news article-summary corpus in five languages and evaluates multilingual and established systems. Results reveal cross-lingual performance differences, including a 66% ROUGE-L improvement from Pointer Generator to M-BERT on Russian.

  • Problem

    Most summarization datasets are English, while no large-scale multilingual document-summarization dataset was available for studying language-specific model performance.

  • Method

    MLSUM constructs a news-based corpus of over 1.5M article-summary pairs in five languages and evaluates extractive, abstractive, supervised, and unsupervised baselines.

  • Results

    66% in terms of ROUGE-L improvement separates Pointer Generator from M-BERT on Russian.

  • Takeaways & Limitations

    Cross-lingual comparative analysis highlights biases and language-dependent performance patterns in summarization systems.

  • Takeaways & Limitations

    Metrics trained from human annotations and several factualness or learned evaluation models were unavailable because they had only been trained on English datasets.

Abstract

from arXiv · show

We present MLSUM, the first large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian, Turkish. Together with English newspapers from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. We report cross-lingual comparative analyses based on state-of-the-art systems. These highlight existing biases which motivate the use of a multi-lingual dataset.

1 Introduction

Summarization requires understanding long documents, selecting relevant content, and writing concise syntheses, but available datasets are overwhelmingly English. MLSUM addresses this gap with a large-scale multilingual dataset, baselines, and cross-lingual analysis.

  • Document summarization combines long-document understanding, relevance discrimination, and short synthesis.
  • Most available summarization datasets are in English, concentrating research efforts on English-language data.
  • Multilingual pretraining and transfer learning enable models to apply task knowledge from pivot languages to other languages.
  • Pivot-language training remains limited because it prevents testing whether models fit one language as well as another, especially for low-resource languages.
  • MLSUM contributes the first large-scale multilingual summarization dataset, strong multilingual abstractive baselines, and comparative cross-lingual analysis.

2 Related Work

Prior multilingual resources support cross-lingual evaluation, but often rely on translated or evaluation-only data, while summarization resources remain largely English and task-specific.

  • Earlier multilingual summarization work included MEAD, genetic-algorithm approaches, and the MultiLing benchmark, which covered 40 languages but offered relatively few examples.
  • Translation-based multilingual summarization for low-resource languages showed only slight baseline improvements and remained far from human performance.
  • Multilingual transfer datasets such as XNLI and MLQA evaluate classification or question answering, but rely on human translations and provide evaluation data.
  • XTREME covers 40 languages and nine tasks, but excludes summarization.
  • XGLUE includes multilingual news-title generation for five languages, but not document summarization.
  • Existing summarization corpora span English news, patents, social media, and other domains, with differing scale, references, and supervision.DUC provides multiple reference summaries but little training data; Gigaword uses headlines generated from article openings rather than human summaries.

3 MLSUM

MLSUM fills the lack of a large-scale non-English document summarization corpus by collecting news article-summary pairs across five languages and documenting their construction and statistics.

  • MLSUM is introduced as the first large-scale multilingual summarization corpus, with more than 1.5 million articles in French, German, Spanish, Turkish, and Russian.
  • The corpus is designed as a multilingual extension of CNN/Daily Mail, using similarly constructed news data and comparable training-sample amounts except for Russian.
  • The paper reports corpus methodology, language statistics, and baseline or state-of-the-art model performance, with article URLs and construction code supporting replication.
  • Newspapers were selected for broad topic coverage, large public archives, and human-written highlights or summaries extractable from webpages.
  • Articles were crawled from 2010 to 2019, discarding articles under 50 words or summaries under 10 words.
  • Russian has ten times fewer training samples than the other MLSUM languages, with the shortest and most abstractive summaries.

4 Models

The evaluation compares extractive and abstractive summarization systems, including simple references, neural sequence-to-sequence models, and multilingual pretrained architectures.

  • Experiments train supervised and unsupervised, extractive and abstractive models separately for each language.
  • The mixed-language model produced no significant performance difference from per-language training.
  • Oracle selects input sentences maximizing ROUGE-L against the reference summary, indicating the extractive upper bound.
  • Random extracts N words randomly, with N fixed to the average summary length, providing an unbiased reference point.
  • Lead-3 selects the first three input sentences, while TextRank extracts the most central sentences using sentence co-similarities.
  • Abstractive systems use encoder-decoder generation, including Pointer-Generator copying out-of-vocabulary tokens and M-BERT-based multilingual generation.

5 Evaluation Metrics

The paper evaluates summaries with ROUGE, METEOR, and novelty, while noting that multilingual neural metrics remain unavailable because existing versions were trained only on English data.

  • Metrics: ROUGE measures n-gram overlap between an evaluated summary and a human reference summary.
  • Metrics: METEOR evaluates machine-generated text using a harmonic mean of unigram precision and recall, weighting recall more heavily.
  • Metrics: Novelty reports the percentage of novel n-grams in generated summaries to assess reliance on extraction and copying.
  • Multilingual evaluation: Existing neural metrics based on question answering, factualness discrimination, or human annotations were trained only on English datasets.

6 Results and Discussion

Experiments compare summarization models across languages and examine how data, language, abstractiveness, and pretraining relate to performance differences. The analyses reveal strong language- and model-specific biases, especially for Russian and English.

  • Analysis factors: The analysis attributes cross-language performance differences to dataset properties and language-specific metric or model biases.Dataset factors include article structure, summary abstractiveness, and data quantity.
  • Analysis factors: Multilingual datasets are needed to study language-specific effects because pivot-language evaluations cannot isolate them.
  • Overall results: The overall system ranking is preserved across ROUGE-L and METEOR, apart from close swaps between Lead 3 and Pointer Generator.
  • Russian: Russian has comparatively low performance, reflecting its highly abstractive corpus and roughly tenfold smaller training set.Pretraining increases Russian Pointer Generator performance by +66% in ROUGE-L when using M-BERT.
  • Abstractiveness: Pointer-generator networks are poorly abstractive, whereas M-BERT is consistently more abstractive, particularly for Russian.The paper links this pattern to copying behavior and exposure to additional texts during pretraining.
  • Dataset abstractiveness: Oracle performance is lower for French and Spanish than for English and German despite similar novelty rates across German, French, and Spanish.Lead-3 ROUGE-L is 35.20 for German and 33.09 for English, versus 19.69 for French and 13.70 for Spanish.
  • Model biases: TextRank performs close to Pointer Generator on English corpora but substantially worse on other languages, suggesting an English bias.The reported TextRank-to-Pointer Generator ratio is 0.85 to 1.21 on English corpora and 0.37 to 0.65 elsewhere.
  • Model biases: M-BERT consistently outperforms Pointer Generator, with larger gains for German than French, potentially reflecting language-specific effects of self-attention.The paper hypothesizes that self-attention may better preserve information in languages with freer word order.

7 Conclusion

The paper presents MLSUM as a large multilingual summarization dataset and uses it to analyze model biases and cross-language performance. It identifies future directions including broader language coverage and multilingual neural metrics.

  • MLSUM contains over 1.5M article-summary pairs in French, German, Russian, Spanish, and Turkish.
  • The dataset complements CNN/Daily Mail English data and supports comparative analysis of state-of-the-art summarization approaches across languages.
  • The experiments highlight biases in existing summarization models and differences in model performance across languages.
  • Future work includes adding Arabic and Hindi and adapting neural evaluation metrics to multilingual summarization.

– FRENCH –

The passage describes Kandahar as a central locus of Afghan political legitimacy, clan influence, and Taliban activity. It also reports worsening insecurity and increasing U.S. operational control in the southern region.

  • The Karzai clan’s regional influence is rooted in Kandahar and shapes national political networks.
  • Kandahar is presented as a historical center of Afghan state formation and a source of political legitimacy distinct from Kabul.
  • Kandahar and the surrounding south are described as the historical heartland of the Taliban insurgency.
  • Taliban influence is associated with growing insecurity, limited employment, intimidation, and the departure of educated residents toward Kabul.
  • The United States increasingly directed operations in Helmand and Kandahar after sending successive reinforcements.

– GERMAN –

Haitis heutige Armut und Krisen werden auf seine konfliktreiche Entstehung und die internationale Behandlung nach der Unabhängigkeit zurückgeführt. Besonders die erzwungenen Reparationszahlungen belasteten das Land langfristig.

  • Die Unabhängigkeit hatte hohe menschliche und wirtschaftliche Kosten: Ein Drittel der Bevölkerung starb in den Kämpfen, und viele Plantagen wurden zerstört.
  • Frankreich und andere Kolonialmächte verweigerten Haiti zunächst die Anerkennung und unterstützten ein Embargo sowie Forderungen nach Reparationszahlungen.
  • Frankreich verlangte 150 Millionen Francs für die Anerkennung Haitis und drohte andernfalls mit einer Invasion und erneuter Versklavung.
  • Haiti nahm Schulden auf und zahlte; die Schuldenlast lähmte bis 1947 die Wirtschaft und legte laut Text den Grundstein für Armut und Korruption.

– SPANISH –

Am Flughafen Barajas führten fehlende Fluglotsen zu einer vorübergehenden Schließung von zwei Start- und Landebahnen. Dadurch kam es trotz späterer Öffnung aller vier Bahnen zu erheblichen Flugverspätungen.

  • Bis zu 60 Minuten Verspätung mit bereits an Bord befindlichen Passagieren meldeten mehrere Fluggesellschaften.
  • Obwohl der Flughafen ab 15.00 Uhr wieder alle vier Start- und Landebahnen betrieb, verzeichneten die Flüge weiterhin erhebliche Verspätungen.
  • 30 Minuten betrugen die durchschnittlichen Verspätungen, nachdem die Abwesenheit von 5 der 18 eingeplanten Fluglotsen zwei Start- und Landebahnen geschlossen hatte.

– TURKISH –

Nicht verbeamtete oder nicht eingestellte Lehrkräfte und Lehramtsbewerber demonstrierten in Ankara gegen Arbeitslosigkeit und die Bildungspolitik. Sie forderten mehr reguläre Stellen und ein transparenteres Auswahlverfahren.

  • Lehramtsbewerber ohne Anstellung demonstrierten in Ankara wegen fehlender Stellen und erhielten Unterstützung von TEKEL-Arbeitern sowie Muharrem İnce.
  • Muharrem İnce kritisierte, dass Schulen ohne Lehrkräfte blieben, während viele Lehramtsabsolventen arbeitslos seien, und verwies auf unterschiedliche Einstellungszahlen nach Fächern.
  • Die Plattform erklärte, jährlich würden Absolventen wegen unzureichender Bildungspolitik arbeitslos; bei ausbleibenden Einstellungen kündigte sie Streiks und einen Hungerstreik an.
  • Die Demonstrierenden forderten reguläre Einstellungen, eine Begrenzung der Lehrerausbildung am tatsächlichen Bedarf und ein transparenteres Verfahren statt KPSS.
Loading 2004.14900v1…