PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehension

Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, Linting Xue

arXiv:2010.12008v3cs.CLcs.AIcs.LG

TL;DR

The efficacy of synthetic QA generation beyond English remains open, while many languages lack manually labeled QA datasets. The paper combines English QA supervision with multilingual MLM to generate target-language QA pairs, reporting large improvements over standard zero-shot baselines while the gap to supervised performance remains.

  • Problem

    Synthetic QA generation is well explored in English, but its efficacy in other languages remains open, and many languages lack manually labeled QA datasets.

    Source: §1 · p. 1

  • Method

    The approach fine-tunes a multilingual generative model using English QA samples and multilingual MLM, then generates target-language QA pairs from unlabeled articles.

    Source: §1 · pp. 1–2

  • Results

    The approach yields large improvements over standard zero-shot reading-comprehension baselines for supported languages.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The approach can be applied to any language supported by the pretrained multilingual generative model.

    Source: §6 · p. 8

    The gap between zero-shot and supervised reading-comprehension performance still remains.

    Source: §6 · p. 8

Abstract

from arXiv · show

We propose a simple method to generate multilingual question and answer pairs on a large scale through the use of a single generative model. These synthetic samples can be used to improve the zero-shot performance of multilingual QA models on target languages. Our proposed multi-task training of the generative model only requires the labeled training samples in English, thus removing the need for such samples in the target languages, making it applicable to far more languages than those with labeled data. Human evaluations indicate the majority of such samples are grammatically correct and sensible. Experimental results show our proposed approach can achieve large gains on the XQuAD dataset, reducing the gap between zero-shot and supervised performance of smaller QA models on various languages.

1 Introduction

Although synthetic QA generation is well explored in English, its efficacy in other languages remains open, while many languages lack manually labeled QA datasets. The paper proposes zero-shot multilingual QA generation using English supervision and multilingual MLM, then uses the synthetic pairs to improve cross-lingual reading comprehension.

1 Introduction

  • The method generates multilingual QA pairs for target-language reading comprehension using English labeled data, avoiding the need to acquire labeled QA datasets in those languages.It can extend to languages supported by the multilingual generative model when unlabeled articles and questions are available.Source: §1 · p. 1
  • Prior work explored question generation and found that large-scale synthetic training data can improve reading-comprehension models, but multilingual efficacy remained open.The paper frames its work against this English-focused background.Source: §1 · p. 1
  • Multi-task fine-tuning combines QA generation with multilingual masked language modeling, which the authors report is crucial for zero-shot generation.The QA task uses English samples, while MLM helps preserve the model's multilingual generation capabilities.Source: §1 · p. 2
  • Synthetic multilingual QA pairs improve zero-shot performance of multilingual reading-comprehension models on multilingual QA tasks.The paper presents this as a contribution of its approach.Source: §1 · p. 2
  • Human evaluations assess whether generated questions are grammatically correct and sensible.The introduction identifies this as an evaluation of generated samples.Source: §1 · p. 2

2 End-to-End Question-Answer Generation and Filtering

The paper fine-tunes multilingual T5 to generate questions and answers from target-language passages, while MLM tasks support multilingual generation. It then filters generated examples and uses synthetic samples to train downstream reading-comprehension models.

2.1 Modeling

  • mT5 is the generative model, an encoder-decoder sequence-to-sequence model based on T5.The model is pretrained on multilingual data.Source: §2.1 · p. 2

2.2 QA Generation Task

  • The QA task conditions on a passage and trains the model to output concatenated question and answer tokens; sampling supplies passages and draws QA pairs from the decoder.The paper prepends “question” and “answer” to their respective token sequences to distinguish them.Source: §2.2 · p. 2

2.3 Masked Language Modeling Task

  • The MLM variant removes sentinel tokens corresponding to non-masked spans from the decoder target sequence, which the authors found improved generated QA quality.The MLM task reconstructs masked spans using a cross-entropy loss.Source: §2.3 · p. 2

2.4 Multi-Task Fine-Tuning

  • Fine-tuning mixes QA-generation examples with mC4 MLM and MLM on TyDiQA questions to retain multilingual generation and further improve question generation.Only SQuAD 1.1 supplies supervised QA training data; the TyDiQA-question MLM is not conditioned on associated passages.Source: §2.4 · p. 3, Figure 2 · p. 3
  • With multi-task fine-tuning, generated examples in five languages are usually in the passage language, with relevant answers and long, non-trivial QA pairs.Without the MLM tasks, Spanish and Arabic examples are primarily in English, although their QA correspondence is semantically well-formed.Source: §2.4 · p. 3

2.5 Decoding and Filtering

  • Roundtrip filtering applies a trained reading-comprehension model to each generated question and passage, retaining the sample only when answer-prediction F1 exceeds a threshold.The procedure compares the generated answer with the model-predicted answer.Source: §2.5 · p. 3

3 Related Work

Prior multilingual QA-generation work differs in whether it generates answers, needs target-language training data or parallel corpora, and evaluates downstream QA performance. The paper contrasts its zero-shot approach with these constraints.

3 Related Work

  • Earlier synthetic-data work shows that large-scale generated training data can improve reading-comprehension models, while related QA-generation efforts were limited to English [7, 42].The paper builds on the modeling and filtering approaches of Shakeri et al. (2020) [42].Source: §3 · p. 3
  • Kumar et al. (2019) and Chi et al. (2019) focus on question-only generation; their approaches require target-language gold samples or parallel sentences and complex pre-training [43, 45].The paper distinguishes its work by evaluating downstream QA improvements rather than only generated-question quality [44, 45].Source: §3 · pp. 3–4
  • Riabi et al. (2020) generate questions but not answers and rely on translated SQuAD to fine-tune, unlike this work’s joint zero-shot QA generation without target-language training data [46].The passage notes that translation quality may also affect generated samples, particularly in low-resource languages.Source: §3 · p. 4

4 Experimental Setup and Results

The experiments evaluate multilingual QA datasets, compare fine-tuning mixtures and model sizes, and assess generated questions automatically and through human ratings. Including MLM tasks improves automatic generation scores, while human evaluations find coherent cross-lingual generation and strong Spanish ratings.

4.1 Datasets

  • The paper uses MLQA’s test split in evaluations and TyDiQA’s Gold Passage task; XQuAD comprises 1,190 professionally translated SQuAD 1.1 dev question-answer samples across ten languages.The paper employed XTREME’s machine-translated SQuAD 1.1 train and dev samples in its experiments, calling them “translate-train.”Source: §4.1 · p. 5

4.2 Generative Model Fine-tuning

  • A 3.7-billion-parameter generative model is fine-tuned with a 10:1 QA Generation-to-MLM mixture, outperforming tested 100:1 and 1000:1 ratios; one model generates across target languages without language-code prompts.Models are fine-tuned for 5,000 steps, and the final checkpoint is used for synthetic-data generation.Source: §4.2 · p. 5

4.3 Automatic Evaluation Results

  • +15 absolute BLEU points at most when MLM tasks are included, compared with QA-generation-only fine-tuning on MLQA test, using mT5-XL [26].MLM on mC4 or TyDiQA questions gives similar BLEU; combining both adds no further BLEU gains, though the combined mixture’s samples appeared more structured and sensible [27].Source: Table 1 · p. 6, §4.3 · p. 6
  • Generation quality improves dramatically with mT5 model size, and the paper uses the 3.7B-parameter XL model for the remaining experiments [28, 30].The comparison covers Base (600M), Large (1B), and XL (3.7B) models in the mixture setting on MLQA [28].Source: §4.3 · p. 6

4.4 Human Evaluations

  • Human raters found the model nearly perfect at matching the input passage’s language, supporting coherent zero-shot cross-lingual generation [35].Four hundred samples across five languages were rated by native speakers on target-language status, grammatical correctness, and sensibility [34, 32].Source: §4.4 · p. 6
  • Spanish samples scored highly across evaluation categories despite Spanish being absent from both the MLM fine-tuning data and SQuAD 1.1 [36].The authors report that the MLM mixture also preserves multilingual capability for languages not included in that fine-tuning task [36].Source: §4.4 · p. 6

5 Application of Synthetic Data to Multilingual Reading Comprehension

This section applies synthetic multilingual QA data to reading-comprehension models and reports gains over zero-shot baselines, with augmentation effects varying by model size.

5 Application of Synthetic Data to Multilingual Reading Comprehension

  • Reading comprehension asks a model to find, in a passage, the span that answers a given question.Source: §5 · pp. 6–7

5.1 Synthetic Data Generation

  • For Arabic, German, Hindi, Russian, and Spanish, the generation pipeline filters synthetic QA pairs to retain passage-supported answers and yields approximately 10–20k samples per language.It starts from 10k Wikipedia paragraphs per language, generates 20 questions per passage, and applies round-trip filtering.Source: §5.1 · p. 7

5.2 RC Model Fine-tuning

  • RC models use mT5 Base, Large, or XL and are fine-tuned on generated samples; when English SQuAD is also used, the mixing ratio is 1.Source: §5.2 · p. 7

5.3 Results

  • +9 absolute F1 points on Russian are observed with the Base model; average F1 improves for every synthetic-data language, with German samples giving the largest average gain (+2.9).The comparison is on XQuAD after augmenting SQuAD en with synthetic samples.Source: §5.3 · p. 7, Table 4 · p. 8
  • With Base, augmentation narrows the zero-shot-to-supervised gap from 7.2 to 4.2 absolute points, but a sizeable gap remains.The supervised comparison uses SQuAD en plus the target-language translate-train dataset.Source: §5.3 · p. 7, Table 4 · p. 8
  • With mT5 Large, average F1 gains reach +1.2, while per-language gains are +5.1 on Russian, +2.2 on Arabic, and +3.4 on Greek.Source: §5.3 · p. 7, Table 5 · p. 8
  • Figure 5 summarizes the scaling effect: augmentation gains shrink as model size grows; XL averages match or fall below zero-shot, though smaller per-language gains remain.Source: §5.3 · p. 7, Figure 5 · p. 8, Table 6 · p. 8

6 Conclusion

The paper presents large-scale synthetic multilingual QA generation to improve zero-shot RC performance. Its results show gains over zero-shot baselines, while the gap to supervised performance remains.

  • The approach generates large-scale synthetic multilingual question-answer data for improving multilingual RC models’ zero-shot performance.Source: §6 · p. 8
  • RC models trained on the synthetic multilingual datasets show large improvements over standard zero-shot baselines, and generation applies to languages supported by the pretrained multilingual model.Source: §6 · p. 8
  • The zero-shot-to-supervised performance gap narrows with synthetic samples but remains; the paper points to improved generation quality as a possible way to reduce it further.Source: §6 · p. 8
Loading 2010.12008v3…