Source-linked AI summary
FlauBERT: Unsupervised Language Model Pre-training for French
Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab
TL;DR
French NLP had benefited less from contextual language-model pre-training and lacked a unified evaluation benchmark. The paper introduces FlauBERT, trained on heterogeneous French data and evaluated across diverse tasks, and reports strong performance against multilingual and cross-lingual models while releasing FLUE for reproducibility.
Problem
French lacked a unified multi-task benchmark comparable to GLUE, while contextual language-model advances had been demonstrated mostly for English.
Method
The paper trains FlauBERT, a French BERT model, on a very large heterogeneous French corpus and evaluates it across diverse NLP tasks.
Results
FlauBERT achieves state-of-the-art results on several French NLP tasks and surpasses multilingual and cross-lingual models.
Takeaways & Limitations
The released FlauBERT models, training pipeline, and FLUE benchmark support reproducible experiments in French NLP.
Abstract
from arXiv · showhide
Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.
1. Introduction
Contextual language representations have driven major NLP advances, but these results have been demonstrated mostly for English. FlauBERT addresses this gap with a French BERT model and a reproducible evaluation framework.
- Contextual embeddings represent each token according to the entire input sequence, unlike word embeddings that assign one fixed vector per wordform.
- FlauBERT is a French Language Understanding via Bidirectional Encoder Representations from Transformers model.
- FlauBERT outperforms multilingual and cross-lingual models on several downstream NLP tasks under similar configurations.
- The paper releases multiple FlauBERT versions, processing and training code, and FLUE, a unified benchmark for French NLP evaluation.
2. Related Work
Pre-trained contextual language models extend earlier fixed word embeddings by incorporating sequence context, but French lacked a unified multi-task evaluation benchmark. Related work spans multilingual models, language-specific models, and French-language benchmark development.
- 2.1. Pre-trained Language Models: Self-supervised language modeling learns useful word representations from unlabeled text, while contextual models make representations depend on the full input sequence.
- 2.1. Pre-trained Language Models: Self-supervised learning uses unlabeled data as a supervision signal.
- 2.1. Pre-trained Language Models: Contextual embeddings address the limitation that a word embedding has only one representation despite words having multiple context-dependent meanings.
- 2.2. Pre-trained Language Models Beyond English: Pre-trained language models have been developed for languages including Portuguese, Japanese, German, Basque, Chinese, and German, alongside multilingual alternatives.
- 2.2. Pre-trained Language Models Beyond English: French already had language-model work, including CamemBERT, while FlauBERT and CamemBERT use different tasks, corpora, and preprocessing pipelines.
- 2.3. Evaluation Protocol for French NLP Tasks: Before this work, French lacked a multi-task benchmark comparable to GLUE for evaluating French NLP systems.
3. Building FlauBERT
FlauBERT is built from a large, heterogeneous French corpus, cleaned and tokenized for BPE-based pre-training with a BERT-style Transformer architecture. The training setup uses masked language modeling, two model sizes, and specialized stabilization choices for the larger model.
- 3.1. Training Data: The corpus combines 24 French sub-corpora from varied sources, spanning formal texts such as Wikipedia and books to Internet crawls.
- 3.1. Training Data: The raw corpus contains 270 GB before preprocessing, while the resulting training corpus contains 71 GB after filtering, normalization, tokenization, and cleaning.
- 3.2. Models and Training Configurations: FlauBERT uses the BERT multi-layer bidirectional Transformer architecture and provides BASE and LARGE model sizes.
- 3.2. Models and Training Configurations: The original BERT pre-training objective combines masked language modeling, which predicts masked tokens, with next sentence prediction.
- 3.2. Models and Training Configurations: Training FlauBERTLARGE with BERTLARGE and RoBERTaLARGE configurations diverged early, motivating methods to address deep-Transformer instability.
- 3.2. Models and Training Configurations: FlauBERT uses sub-word masking and a 50K BPE vocabulary, with French Moses tokenization applied before BPE encoding.
4. FLUE
FLUE is a French NLP benchmark built from tasks spanning varied domains, difficulty levels, formality, and training-set sizes. It combines cross-lingual and French-specific evaluations, including classification, paraphrasing, inference, parsing, and word sense disambiguation.
- FLUE compiles French NLP tasks across different domains, difficulty levels, formality degrees, and amounts of training data.
- Three of its six tasks—text classification, paraphrase, and natural language inference—come from cross-lingual datasets.This design supports comparisons between monolingual pre-training and future cross-lingual models.
- Text Classification: The benchmark includes sentiment classification using French Amazon reviews labeled positive or negative after removing three-star ratings.Accuracy is reported on the test set.
- Paraphrasing: French paraphrase evaluation uses PAWS-X pairs designed to distinguish semantic equivalence despite high lexical overlap.The French data are evaluated with test-set accuracy; training examples are machine-translated, while development and test sets are human-translated.
- Natural Language Inference: Natural language inference uses premise–hypothesis pairs labeled entailment, contradiction, or neutral in the French portion of XNLI.Each language’s development and test sets contain 7,500 human-annotated examples.
- Parsing and Word Sense Disambiguation: The remaining evaluations cover syntactic parsing and French word sense disambiguation for verbs and nouns.The parsing data include constituency and dependency trees with part-of-speech tags; the WSD tasks use FrenchSemEval and a modified multilingual WSD dataset.
5. Experiments and Results
FlauBERT is evaluated against multilingual and French models across FLUE tasks using standardized fine-tuning setups. Results generally favor French pre-trained models, with performance varying by task and model size.
- Text Classification: FlauBERT and CamemBERT outperform mBERT on French text classification, while FlauBERTLARGE achieves the best results across books, DVD, and music.FlauBERTBASE is moderately better than CamemBERT on books but lower on DVD and music.
- Paraphrasing: On PAWS-X, monolingual French models perform only slightly better than mBERT, likely because high lexical overlap tests word order and syntactic sensitivity.The dataset’s construction may allow multilingual models to capture relevant features as effectively as monolingual models.
- Natural Language Inference: On XNLI, French models outperform mBERT; FlauBERTLARGE is moderately better than CamemBERT, while both outperform XLM-RBASE but not XLM-RLARGE.
- Constituency Parsing and POS Tagging: FlauBERTBASE outperforms CamemBERT by 0.7 absolute F1 in constituency parsing and obtains the best published single-model result.The CamemBERT–FlauBERT ensemble improves over FlauBERT by 0.4 absolute F1 on development data and 0.2 on test data.
- Constituency Parsing and POS Tagging: Large-scale pre-trained language models achieve POS tagging accuracy of 98.1-98.2, exceeding models without pre-training or using fastText embeddings at 97.5-97.7.FlauBERTLARGE marginally improves development performance but does not match FlauBERTBASE on the test set.
- Word Sense Disambiguation: For verb sense disambiguation, BERT-based models establish the new state of the art, with the best results achieved by CamemBERT and FlauBERTLARGE.For noun sense disambiguation, FlauBERTLARGE achieves the best scores, more than 1 point above mBERT.
6. Conclusion
The paper presents FlauBERT as a French pre-trained language model achieving strong results across French NLP tasks. It also releases reproducibility resources, including preprocessing and training scripts and the FLUE benchmark.
- FlauBERT achieves state-of-the-art results on several French NLP tasks, surpassing multilingual and cross-lingual models.
- FlauBERT remains competitive with CamemBERT despite being trained on almost twice as fewer text data.
- The authors release FlauBERT together with preprocessing and training scripts to support a reproducible pipeline.
- FLUE provides a general benchmark for evaluating French NLP systems, and FlauBERT is supported by Hugging Face’s transformers library.
A.1 Details on our French text corpus
The French training corpus combines 24 sub-corpora from diverse sources, including web, news, institutional, literary, and collaboratively produced text. It includes both monolingual resources and datasets collected from OPUS and other sources.
- The corpus comprises 24 sub-corpora gathered from different sources and spanning diverse topics and writing styles.
- Its sources include web crawls, news, parliamentary proceedings, news commentary, and text from the European Parliament.
- The corpus includes OPUS resources such as OpenSubtitles and EU Bookshop publications.
- Other sub-corpora include MultiUN, GIGA, DGT, Global Voices, TED Talks, and the European constitution.
- Wikimedia resources contribute Wikipedia, Wikisource, Wikinews, Wiktionary, Wikiversity, Wikibooks, Wikiquote, and Wikivoyage text.
- Additional sources include Project Gutenberg ebooks, Enron email data, patent documents, and Le Monde articles collected from 1987 to 2003.