PaperlayerView exact revision on arXiv

Source-linked AI summary

WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

Faisal Ladhak, Esin Durmus, Claire Cardie, Kathleen McKeown

arXiv:2010.03093v1cs.CL

TL;DR

Cross-lingual abstractive summarization lacks large, high-quality multilingual resources for evaluation. WikiLingua addresses this gap with an 18-language dataset of image-aligned WikiHow article-summary pairs and a direct method using synthetic data and machine-translation pre-training. The method outperforms baselines without requiring translation at inference time.

  • Problem

    Cross-lingual abstractive summarization has limited large, high-quality multilingual resources because aligned datasets are costly and difficult to create.

  • Method

    WikiLingua extracts article-summary pairs from WikiHow across 18 languages, aligns steps through shared illustrative images, and pre-trains direct summarization with synthetic data and machine translation.

  • Results

    The proposed direct method outperforms existing baselines without relying on translation at inference time.

  • Takeaways & Limitations

    WikiLingua provides a large benchmark for cross-lingual and multilingual abstractive summarization research.

  • Takeaways & Limitations

    The paper’s initial study focuses on generating English summaries from non-English articles and leaves multi-source and any-language-to-any-language summarization for future work.

Abstract

from arXiv · show

We introduce WikiLingua, a large-scale, multilingual dataset for the evaluation of crosslingual abstractive summarization systems. We extract article and summary pairs in 18 languages from WikiHow, a high quality, collaborative resource of how-to guides on a diverse set of topics written by human authors. We create gold-standard article-summary alignments across languages by aligning the images that are used to describe each how-to step in an article. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset. We further propose a method for direct crosslingual summarization (i.e., without requiring translation at inference time) by leveraging synthetic data and Neural Machine Translation as a pre-training step. Our method significantly outperforms the baseline approaches, while being more cost efficient during inference.

1 Introduction

WikiLingua addresses the scarcity of large, high-quality multilingual resources by introducing an 18-language benchmark built from collaboratively written WikiHow guides. It also evaluates existing methods and proposes direct cross-lingual summarization using synthetic data and machine-translation pre-training.

  • Motivation: Cross-lingual abstractive summarization has lacked large, high-quality datasets because creating aligned summaries through crowdsourcing is costly and subjective.Human annotators must read, comprehend, condense, and paraphrase entire articles while selecting salient content.
  • Dataset: WikiLingua contains collaboratively written how-to guides with gold-standard summaries across 18 languages.Articles cover diverse procedural topics, and each step includes a summary, detailed paragraph, and illustrative image.
  • Alignment: The dataset aligns cross-language steps by matching images that are specific to each step and shared across languages.This handles differences in step ordering across language versions.
  • Scale: 141,457 unique English articles form the final dataset, with an average of 42,783 aligned articles in each other language.The authors identify WikiLingua as the largest dataset with parallel articles and summaries for cross-lingual abstractive summarization at the time.
  • Methods and evaluation: The proposed direct cross-lingual method uses synthetic data and machine translation as pre-training, then outperforms existing baselines without translation at inference.The approach is also described as more cost efficient during inference.

2 Data Collection and Statistics

WikiLingua is constructed from human-written, reviewed WikiHow procedures across 18 languages, producing aligned article-summary pairs by matching step images. Its scale and topic diversity substantially exceed prior cross-lingual resources.

  • Source data: WikiHow provides human-written and reviewed how-to guides covering diverse multi-step procedural topics.Each method contains steps with unique images, one-sentence summaries, and detailed instructional paragraphs.
  • Cross-lingual alignment: The dataset collects guides in all 18 available WikiHow languages and aligns corresponding steps using illustrative images.The images are unique to individual steps and shared across language versions, supporting high-quality alignment.
  • Pair construction: Step summaries and paragraphs are merged for each method to create article-summary pairs in every language.The resulting language-specific pairs are counted by their alignment with English articles.
  • Statistics: Turkish has the fewest parallel article-summary pairs with English but still exceeds any existing cross-lingual dataset by an order of magnitude.This comparison highlights the dataset’s scale even for its least represented parallel language.
  • Statistics: WikiLingua compares datasets using language coverage, average summaries per language, and average summary and article token lengths.These statistics are reported in Table 1.

3 Existing Multilingual Abstractive Summarization Datasets

Existing multilingual summarization datasets provide limited support for cross-lingual evaluation: some lack parallel articles and summaries, while others use small or indirect summaries. WikiLingua is larger and covers a wider range of topics.

  • Prior datasets: MultiLing contains summaries for Wikipedia articles in several languages but lacks parallel articles or summaries, limiting its use for cross-lingual summarization.Its collections include 30 articles per language in MultiLing’13 and 30 additional evaluation documents in MultiLing’15.
  • Prior datasets: Global Voices descriptions are designed to attract social-media clicks rather than summarize article content, so they provide lower coverage than good summaries.A small crowdsourced English-summary set was created for 15 languages to address this issue.
  • Comparison: Existing multilingual datasets are limited in size compared with WikiLingua, while WikiLingua covers a wide range of topics.Its average number of articles per language is two orders of magnitude larger than Global Voices.
  • WikiLingua statistics: WikiLingua includes 141,457 English article-summary pairs in its dataset statistics.Table 2 defines parallel articles as those having an aligned article in English.
  • Evaluation setup: The benchmark organizes examples into train, validation, and test splits for each language.Table 3 reports the number of examples in each split per language.

4 Cross-lingual Experiments

The experiments compare baseline and direct cross-lingual summarization methods across five languages, using mBART and several translation-based strategies. The proposed direct approach uses synthetic data and machine-translation pre-training to avoid translation at inference time.

  • Experiments generate English summaries from articles in Spanish, Russian, Turkish, and Vietnamese, covering three language families.
  • mBART provides the shared multilingual sequence-to-sequence foundation for monolingual and cross-lingual summarization fine-tuning.The model uses shared sub-word vocabulary, encoder, and decoder components across 25 languages.
  • 4.1 Baselines: The baselines include lead-3, summarize-then-translate, translate-then-summarize, round-trip translation, and an oracle using gold translations.The translation-based systems differ in whether translation occurs before or after summarization and whether translations are machine-generated or gold.
  • 4.1 Baselines: Two-step systems require translation and summarization sequentially at inference time, increasing recurring latency and monetary cost.
  • 4.2 Direct Cross-lingual Summarization: Direct cross-lingual summarization trains on source-language articles and target-language summaries, with synthetic translated training data added to address limited cross-lingual supervision.The synthetic-data variant shifts translation cost to training time as a one-time cost.
  • 4.2 Direct Cross-lingual Summarization: The proposed pre-training strategy first fine-tunes mBART for document-level machine translation, then fine-tunes it for cross-lingual summarization.

5 Results and Analysis

The experiments compare translation-based and direct cross-lingual summarization methods. Synthetic-data and machine-translation pre-training improve direct summarization, while human evaluation confirms stronger content overlap for the proposed model.

  • Baseline comparisons: Source-language data availability determines whether Trans-Sum or Sum-Trans performs better.Trans-Sum works significantly better with limited source-language data; as source data grows, the performance gap decreases, and Sum-Trans may become more cost effective.
  • Baseline comparisons: 22% of the performance loss from translation errors is recovered by round-trip translation in Trans-Sum-R.Trans-Sum-R is the strongest baseline and consistently outperforms Trans-Sum.
  • Direct cross-lingual summarization: DC performs worse than translate-then-summarize baselines for every language except Spanish.This indicates that direct cross-lingual summarization is difficult and requires more data, even with pre-trained mBART.
  • Direct cross-lingual summarization: Synthetic data substantially improves DC, especially for Turkish and Vietnamese, bringing them on par with the best baseline.DC+Synth remains preferable because it achieves similar performance while being much more cost effective at inference.
  • Direct cross-lingual summarization: Machine-translation pre-training further improves DC+Synth for all languages.DC+Synth+MT significantly outperforms all baselines for Spanish, Russian, and Vietnamese; for Turkish, it is statistically equivalent to Trans-Sum-R.

6 Related Work

Prior cross-lingual summarization work relies mainly on summarize-then-translate or translate-then-summarize pipelines and has limited direct-summarization research. Existing approaches face data, translation-quality, and error-propagation constraints, motivating WikiLingua’s broader benchmark and direct method.

  • Abstractive summarization: Most abstractive summarization research has focused on monolingual English settings, alongside advances from neural and pre-trained language models.The related work traces progression from attention-based convolutional models to recurrent decoders, extractive-abstractor systems, and pre-trained models.
  • Cross-lingual abstractive summarization: Summarize-then-translate and translate-then-summarize are established approaches for cross-lingual summarization.Summarize-then-translate translates only the summary, while translate-then-summarize translates the article before summarization.
  • Cross-lingual abstractive summarization: Summarize-then-translate is less expensive and may reduce translation error propagation, but it requires substantial source-language summarization data.Translate-then-summarize instead depends on accurate translation and sufficient target-language summarization data.
  • Cross-lingual abstractive summarization: Round-trip translation creates noisy articles paired with clean summaries so models can account for translation noise.Ouyang et al. use this strategy as a translate-then-summarize variant.
  • Direct cross-lingual summarization: Direct cross-lingual summarization has received limited prior attention, including zero-shot headline generation and sentence summarization.The paper describes its approach as simpler and focused on a different summarization task.
  • Direct cross-lingual summarization: A prior multi-task synthetic-data approach performed worse than all baselines except lead on this data.The authors suspect the difference may relate to the smaller synthetic dataset and leave pre-trained-mBART multi-task training for future work.

7 Conclusion

The paper presents WikiLingua as a benchmark for cross-lingual and multilingual abstractive summarization and proposes a direct end-to-end method. The dataset is intended to support further research, including multi-source and multilingual summarization.

  • WikiLingua is presented as a benchmark dataset for cross-lingual and multilingual abstractive summarization.
  • The proposed end-to-end direct method achieves significantly better performance than baselines while being more cost effective for inference.
  • The dataset opens directions for multi-source cross-lingual summarization and multilingual summarization across language pairs.These directions include summarizing from multiple source languages and mapping any language to any other language within a language set.

A.1 Reproducibility

The experiments use Fairseq and mBART-based settings with specified hardware, stopping criteria, inference beam size, and official ROUGE evaluation.

  • Fairseq is used for all experiments, with hyperparameters following Lewis et al. (2019b).The authors modified existing mBART code to support monolingual summarization.
  • Models are trained on four Nvidia Tesla V100 GPUs, 96 CPU cores, and 693 GB of RAM.
  • Training stops when validation loss fails to improve for two epochs, using the checkpoint with the best validation loss for inference.
  • Table 4 models use the same approximately 680M-parameter pre-trained mBART architecture and hyperparameters, with beam size five at inference.ROUGE scores are computed using the official ROUGE script.

A.2 Splitting English Data

The English test split includes parallel articles and additional topic-overlapping articles to reduce unfair cross-lingual evaluation from shared content.

  • Parallel English articles corresponding to test articles in any of four languages are mapped into the English test set.
  • Topic-overlapping English articles are also included because multiple articles may cover the same topic with overlapping content.

A.3 Data Statements

WikiLingua is collected from multilingual WikiHow articles and aligned through step-level images, with broad topical coverage and documented data limitations. Human-evaluation examples show preferences for both the proposed system and baselines.

  • Data Collection: English WikiHow articles are linked to corresponding articles in 17 other languages and aligned using each step’s illustrative image.The images are treated as unique to individual how-to steps.
  • Language Coverage: The dataset contains articles in 18 named languages, but information about language varieties is unavailable.
  • Data Quality: WikiHow articles are collaboratively written, edited, and reviewed, while many non-English articles receive bilingual and international translation-team review.On average, each article and summary is written and edited by 23 people and further reviewed by 16 people.
  • Topic Coverage: The articles span 19 broad categories, including health, travel, education, and personal care and style.
  • Human Evaluation Examples: Human annotators preferred DC+Synth+MT in two examples and preferred baselines in two others.The evaluated systems were Trans-Sum, Trans-Sum-R, and DC+Synth+MT.
Loading 2010.03093v1…