PaperlayerView exact revision on arXiv

Source-linked AI summary

FQuAD: French Question Answering Dataset

Martin d'Hoffschmidt, Wacim Belblidia, Tom Brendlé, Quentin Heinrich, Maxime Vidal

arXiv:2002.06071v2cs.CLcs.AIcs.LG

TL;DR

French Reading Comprehension lacked a native dataset, leaving a resource gap for evaluating French models and cross-lingual approaches. FQuAD supplies manually annotated questions and answers on French Wikipedia articles, and its best model, CamemBERTLARGE, surpasses reported human performance on both metrics.

  • Problem

    French Reading Comprehension models had not been evaluated because no French dataset was available.

    Source: §2.3 · p. 3

  • Method

    FQuAD manually annotates questions and answers on high-quality French Wikipedia articles, expanding from 25k+ questions to 60k+ across two versions.

    Source: §9 · p. 12

  • Results

    CamemBERTLARGE achieves 92.2% F1-score and 82.1% Exact Match, surpassing reported human performance on both metrics.

    Source: §9 · p. 12

  • Takeaways & Limitations

    FQuAD provides a native French resource for evaluating French Question Answering models and cross-lingual approaches.

    Source: §1 · p. 2, §8 · p. 11

    FQuAD does not yet include adversarial questions; extending the dataset with them is left for future work.

    Source: §9 · p. 12

Abstract

from arXiv · show

Recent advances in the field of language modeling have improved state-of-the-art results on many Natural Language Processing tasks. Among them, Reading Comprehension has made significant progress over the past few years. However, most results are reported in English since labeled resources available in other languages, such as French, remain scarce. In the present work, we introduce the French Question Answering Dataset (FQuAD). FQuAD is a French Native Reading Comprehension dataset of questions and answers on a set of Wikipedia articles that consists of 25,000+ samples for the 1.0 version and 60,000+ samples for the 1.1 version. We train a baseline model which achieves an F1 score of 92.2 and an exact match ratio of 82.1 on the test set. In order to track the progress of French Question Answering models we propose a leader-board and we have made the 1.0 version of our dataset freely available at https://illuin-tech.github.io/FQuAD-explorer/.

1 Introduction

French Reading Comprehension lacks native annotated resources, while language-specific datasets are costly to collect. The paper introduces FQuAD and evaluates French and multilingual models, including cross-lingual approaches.

1 Introduction

  • The shortage of native-language annotated datasets is identified as a key reason French Question Answering development is slower.The introduction also notes that translated and cross-lingual approaches have not reached English-comparable performance on other languages.Source: §1 · p. 1
  • FQuAD provides French native questions and answers in two versions, with 25,000+ samples in FQuAD1.0 and 60,000+ in FQuAD1.1.The additional 35k+ samples use more demanding guidelines to increase data and task complexity.Source: §1 · pp. 1–2
  • The experiments fine-tune BERT-based Question Answering models, covering French CamemBERT and FlauBERT as well as multilingual mBERT and XLM-RoBERTa.The stated aim is to compare how multilingual models perform on native datasets beyond English.Source: §1 · p. 2
  • The paper tests zero-shot cross-lingual transfer on FQuAD and models fine-tuned on a French-translated SQuAD1.1 dataset.These experiments are intended to assess the two approaches on a native French dataset.Source: §1 · p. 2

2 Related Work

The related work reviews English Reading Comprehension datasets, native datasets in other languages, cross-lingual approaches, and language models. It highlights the lack of a French dataset for evaluating Reading Comprehension models.

2.1 Reading Comprehension in English

  • English Reading Comprehension resources include datasets for span-based QA, unanswerable questions, conversational QA, information-seeking dialogue, and multi-document reasoning.The section describes SQuAD2.0 as introducing unanswerable adversarial questions, CoQA and QuAC as conversational settings, and HotPotQA as multi-hop reasoning.Source: §2.1 · p. 2

2.2 Reading Comprehension in other languages

  • Native Reading Comprehension datasets exist in Russian, Chinese, and Korean, while the cited French PIAF project contains 3,835 question-answer pairs.The related work characterizes native datasets outside English as rare.Source: §2.2 · p. 2
  • Cross-lingual QA uses zero-shot transfer or translated training data; the cited XQuAD and MLQA datasets do not cover French.The translation approach described for French reaches 76.7/61.8 % F1/EM on a small translated corpus.Source: §2.2 · pp. 2–3

2.3 Language modeling for Reading Comprehension

  • Pretrained multilingual models aim to address English-only language-model coverage, but French Reading Comprehension performance had not been evaluated because no French dataset was available.The section notes that multilingual models perform reasonably across languages but do not appear to outperform language-specific models.Source: §2.3 · p. 3
  • Table 1 benchmarks available Reading Comprehension datasets, including FQuAD, which the paper presents in the following sections.The related work also notes the limited availability of French resources and recent French pretrained models.Source: Table 1 · p. 3, §2.3 · p. 3

3 Dataset Collection

FQuAD is collected in two stages from French Wikipedia articles, with article-level splits and manually annotated question-answer pairs. The expanded FQuAD1.1 asks annotators for more complex questions; adversarial samples are left for future work.

3.1 Paragraphs collection

  • The collection follows SQuAD1.1 guidelines, using 1,769 quality French Wikipedia articles and random article-level splits to avoid possible biases.The FQuAD1.0 and FQuAD1.1 collections use 145 and 326 articles, respectively.Source: §3 · p. 3, §3.1 · p. 3

3.2 Question and answer pairs collection

  • For each paragraph, annotators create 3–5 questions whose answers are the smallest spans containing the answer, spending about one minute per pair.The interface is described as encouraging workers to write difficult questions.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 Additional answers collection

  • Development and test questions receive two additional answers each, giving three answers per question to reduce annotation bias and indicate human performance.Workers spend about 30 seconds on each additional answer.Source: §3.3 · p. 4

3.4 FQuAD1.0

  • FQuAD1.0 contains 26,108 collected question-answer pairs, with its article, paragraph, and question counts reported in Table 2.A complete annotated paragraph is shown as an example.Source: §3.4 · p. 4, Table 2 · p. 4, Figure 2 · p. 4

3.5 FQuAD1.1

  • FQuAD1.1 contains 62,003 questions, with workers asked to vary question styles and types to increase difficulty.Its training, development, and test sets draw from 271, 30, and 25 articles, respectively.Source: §3.5 · pp. 4–5, Table 3 · p. 5
  • FQuAD1.1 extends the initial dataset with more demanding question-writing guidelines, while retaining the same additional-answer collection process.The expansion is reported as 62,003 questions overall.Source: §3.5 · pp. 4–5

3.6 Adversarial samples

  • The dataset does not contain adversarial samples like those in SQuAD2.0; adding them is left for future work.Source: §3.6 · p. 5

4 Dataset Analysis

The dataset analysis examines answer and question types, then characterizes question–answer differences in FQuAD.1.1. It reports that “What” questions account for 47.8% of the corpus and manually categorizes sampled question–answer relationships.

4 Dataset Analysis

  • The analysis combines answer-type patterns, question-type frequencies, and question–answer relationship categories to characterize dataset diversity.Source: §4 · p. 5

4.1 Answer analysis

  • Answer types are classified using regular expressions, spaCy entity extraction, and rules for remaining proper nouns and phrase categories.The resulting categories include dates and other numbers, persons, locations, common nouns, verbs, adjectives, and an “other” label.Source: §4.1 · p. 5, Table 4 · p. 5

4.2 Question analysis

  • “What” questions represent 47.8% of the corpus, while the annotation process yields a wide range of question types.The authors suggest this category includes both English “What” and “Which” formulations and may reflect annotator or French-language bias.Source: §4.2 · p. 5
  • The authors’ intuition is that any annotator bias in question wording also occurs at inference because it originates in native French structure.Source: §4.2 · p. 5

4.3 Question-answer differences

  • Question–answer differences are analyzed by manually labeling six randomly sampled questions per development-set article, allowing samples to belong to multiple categories.Categories adapted from include synonymy, world knowledge, syntactic variation, and multiple-sentence reasoning.Source: §4.3 · p. 5

5 Dataset Evaluation

The evaluation defines French-adjusted scoring, measures human agreement, and compares FQuAD1.1 with SQuAD1.1. Human F1 is similar across the datasets, while answer length and the number of reference answers help contextualize their Exact Match difference.

5.1 Evaluation metrics

  • For multiple-answer questions, F1 is the maximum overlap score across ground-truth answers; French scoring also ignores specified French articles.EM measures exact prediction matches, while F1 averages token overlap; the French normalization ignores le, la, les, l’, du, des, au, aux, un, and une.Source: §5.1 · p. 6

5.2 Human performance

  • Human performance is estimated by rotating each of three answers as the prediction against the other two as ground truth, then averaging the three runs.Source: §5.2 · p. 6
  • On the FQuAD1.1 test set, human scores are 91.2% F1 and 75.9% EM; its 3,405 new questions score 74.1% EM versus 78.4% on FQuAD1.0.The reported FQuAD1.0 test scores are 92.1% F1 and 78.4% EM; the authors take the new-question EM gap as evidence of greater difficulty.Source: Table 7 · p. 6, §5.2 · p. 6

5.3 Comparing FQuAD1.1 and SQuAD1.1

  • Human test scores are 91.2% F1 for both FQuAD1.1 and SQuAD1.1, but their EM scores are 75.9% and 82.3%, respectively.The authors describe the Exact Match difference as 6%.Source: §5.3 · p. 6, §5.2 · p. 6
  • Common nouns are the most frequent answer type in both datasets: 26.6% for FQuAD1.1 and 31.8% for SQuAD1.1; FQuAD1.1 has higher proportions of most structured entities.The exception among the listed structured entities is Date, for which FQuAD1.1 scores lower.Source: Table 8 · p. 7, §5.3 · p. 7
  • FQuAD1.1 answers average 4.24 tokens, compared with 2.72 for SQuAD1.1, and are generally longer, making high Exact Match harder to reach on FQuAD.Source: Figure 3 · p. 7, §5.3 · p. 7
  • Exact Match quickly declines as answer length increases in both datasets, whereas F1 is less affected and similar across them.The authors suggest the answer-length distributions may explain part of the EM difference, but do not seem to affect F1.Source: Figure 4 · p. 7, §5.3 · p. 7
  • SQuAD1.1 and SQuAD2.0 average 4.8 human answers per question, compared with 3 for FQuAD1.1, contributing to SQuAD1.1’s higher human EM.The authors note that more reference answers make it more likely a prediction matches an expected answer.Source: §5.3 · pp. 7–8

6 Experiments

The experiments evaluate FQuAD and fine-tuned models through native French and cross-lingual settings. They use a shared training setup and include model comparisons, performance analyses, a learning curve, and an assessment involving PIAF.

6.1 Experimental set-up

  • Experiments use three epochs, a 3.0 · 10^-5 learning rate with 6% warm-up, batch sizes of 8 for base and 4 for large models, and AdamW.Runs use HuggingFace transformers on a single V100 GPU.Source: §6.1 · p. 8

6.2 Native French Reading Comprehension

  • CamemBERT, FlauBERT, mBERT, and XLM-RoBERTa are fine-tuned on FQuAD1.0 and FQuAD1.1 training sets and evaluated on FQuAD1.1 development and test sets.Source: §6.2 · p. 8
  • The best model’s F1 and Exact Match predictions are analyzed across question and answer categories to examine performance by type.Source: §6.2 · p. 8
  • A CamemBERTBASE learning curve reports test-set EM and F1 as training size increases, addressing how much data question answering may require.Source: §6.2 · p. 8
  • The PIAF experiments evaluate FQuAD1.0-trained CamemBERT on PIAF’s 3,885 training samples and test whether combining PIAF with FQuAD1.0 improves FQuAD1.1 test scores.Source: §6.2 · p. 8

6.3 Cross-lingual Reading Comprehension

  • Zero-shot experiments fine-tune multilingual models on English SQuAD1.1 and evaluate on FQuAD1.1 development data, with the reverse FQuAD-to-SQuAD evaluation also performed.The work notes that French is not supported by the cited cross-lingual benchmark.Source: §6.3 · p. 8
  • For translated-data experiments, CamemBERT is fine-tuned on about 40.7k French-translated SQuAD1.1 pairs and evaluated on native FQuAD1.1 development data.Translated answers that do not align with translated paragraph start/end positions are discarded.Source: §6.3 · pp. 8–9

7 Results

The experiments compare native French and cross-lingual reading-comprehension setups. CamemBERTLARGE achieves the strongest reported native result, while translated-data training and zero-shot transfer produce distinct performance patterns.

7.1 Native French Reading Comprehension

  • 92.2% F1 and 82.1% EM: CamemBERTLARGE trained on FQuAD1.1 achieves the highest experimental test scores and exceeds human performance on both metrics.The comparison table covers models trained on FQuAD1.1 and evaluated on its test and development sets.Source: Table 9 · p. 9, §7.1 · p. 9
  • XLM-RLARGE leads multilingual models at 89.5% F1 and 79% EM on FQuAD1.1-test, while monolingual models perform stronger overall.XLM-RLARGE nevertheless surpasses CamemBERTBASE on test and development sets.Source: §7.1 · p. 9
  • Structured Date, Numeric, and Location answers and How many, Where, and When questions perform well, while How/Why questions and Verb exact matches are weaker.The passage suggests longer, wordier answers may contribute to the weaker results for How/Why and Verb answers.Source: §7.1 · pp. 9–10
  • 88.4% F1 and 78.4% EM are the learning-curve maxima; both metrics improve quickly on the first 10,000 samples before flattening.CamemBERTBASE exceeds human Exact Match after training on 30,000 or more samples.Source: Figure 5 · p. 10, §7.1 · p. 10
  • FQuAD-trained models score lower on PIAF than on FQuAD1.1-test; the authors identify PIAF's lack of multiple answers per question as one reason for the gap.Table 13 reports CamemBERT models trained on FQuAD1.0-train and evaluated on PIAF.Source: §7.1 · p. 10, Table 13 · p. 10

7.2 Cross-lingual Reading Comprehension

  • 86.1% F1 and 73.2% EM: XLM-RLARGE trained on SQuAD1.1 transfers zero-shot to FQuAD1.1-dev, with drops of 3.0% F1 and 5.7% EM versus native training.The zero-shot experiments also evaluate transfer in the reverse direction, from FQuAD1.1 training to SQuAD1.1 development.Source: Table 14 · p. 11, Translated Reading Comprehension · p. 11

Translated Reading Comprehension

  • CamemBERTBASE trained on translated SQuAD1.1 reaches 81.8% F1 and 67.8% EM, while CamemBERTLARGE reaches 87.5% F1 and 73.9% EM.Both translated-data results are below their native-data counterparts; adding translated samples to native data brings no significant improvement.Source: Translated Reading Comprehension · p. 11

8 Discussion

The discussion presents FQuAD as a resource for French reading-comprehension research and cross-lingual evaluation. It reports strong model results while identifying differences between native, translated, and zero-shot settings.

8 Discussion

  • FQuAD supports evaluation of French question answering and cross-lingual transfer, areas for which the authors say quality annotated data in languages beyond English is important.The discussion also notes that monolingual models' advantage over multilingual models is narrower than expected for reading comprehension.Source: §8 · p. 11

8.1 Monolingual vs. multilingual language models

  • For comparable model sizes, monolingual models outperform multilingual models overall, but XLM-RLARGE exceeds CamemBERTBASE and surpasses human Exact Match on FQuAD1.1-test.CamemBERTLARGE exceeds human performance on both metrics across development and test datasets.Source: §8.1 · pp. 11–12

8.2 Translated Reading Comprehension

  • On FQuAD1.1-dev, translated-data CamemBERTBASE scores 81.8/67.8% F1/EM versus 88.1/78.1% with native data; adding translated samples does not improve native evaluation.The authors conclude that translation-based approaches reach a performance ceiling with translated data, while noting possible translation inaccuracies.Source: §8.2 · p. 12

8.3 Cross-lingual Reading Comprehension

  • Zero-shot transfer works in both directions without target-language labels, but results are better on SQuAD than FQuAD; structural differences, FQuAD difficulty, or English-heavy pretraining may explain the gap.Training multilingual models on both FQuAD1.1 and SQuAD1.1 is proposed for future work.Source: §8.3 · p. 12

9 Conclusion

FQuAD provides a manually annotated native-French reading-comprehension dataset, expanded from 25k+ to 60k+ questions. Its best reported model surpasses the stated human performance, and portions of the dataset are publicly available to support French NLP research.

9 Conclusion

  • FQuAD is a native-French Reading Comprehension dataset built from high-quality Wikipedia articles, with 60k+ manually annotated questions across its two versions.FQuAD1.0 contains 25k+ questions; FQuAD1.1 reaches 60k+ and adds answers to the development and test sets for evaluation.Source: §9 · p. 12
  • 91.2% F1-score and 75.9% Exact Match on test; 92.1% F1-score and 78.3% Exact Match on development, with human scores comparable to SQuAD1.1.Source: §9 · p. 12
  • 92.2% F1-score and 82.1% Exact Match: CamemBERTLARGE surpasses established human performance on both metrics.The experiments also report that monolingual models of comparable sizes perform better than multilingual models.Source: §9 · p. 12
  • The FQuAD1.0 training set and FQuAD1.1 development set are publicly available to foster research in French NLP.Source: §9 · p. 12
  • Extending the dataset to adversarial questions, as in SQuAD2.0, is left for future work.Source: §9 · p. 12

A Example model predictions

The examples pair article passages with questions and answers. They cover Brexit-related statements and a climate report’s discussion of warming, emissions, and energy scenarios.

A Example model predictions

  • The Brexit example describes debate over a possible second referendum, including statements by Tony Blair and Dominic Grieve about the referendum and a no-deal exit.Source: A Example model predictions · p. 15
  • Questions about the Brexit passage ask what event was discussed, how many Conservative colleagues were expelled, what the referendum concerned, and when that occurred.Source: A Example model predictions · p. 15
  • The climate-report example describes warming projections and emissions reductions associated with limiting warming to 1.5 °C or below 2 °C.Source: A Example model predictions · p. 15
  • Questions on the climate passage cover the report’s subject, greenhouse gases, emissions targets, and projected renewable-energy shares.Source: A Example model predictions · p. 15

Related papers

Loading 2002.06071v2…