PaperlayerView exact revision on arXiv

Source-linked AI summary

CamemBERT: a Tasty French Language Model

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, Benoît Sagot

arXiv:1911.03894v3cs.CL

TL;DR

High-performing pretrained language models were largely English-focused or multilingual, limiting monolingual use in other languages. The paper trains the French RoBERTa-based CamemBERT, varies pretraining corpora, and evaluates four downstream tasks. CamemBERT reaches or improves the state of the art across those tasks, while experiments indicate that web-crawled and relatively small datasets can perform strongly.

  • Problem

    High-performing pretrained language models were largely trained on English or multilingual data, while large-scale monolingual models for other languages remained limited.

  • Method

    The paper trains French CamemBERT using the RoBERTa architecture and OSCAR web-crawled data, evaluates four downstream tasks, and compares corpus sizes and domains.

  • Results

    CamemBERT reaches or improves the state of the art in POS tagging, dependency parsing, NER and NLI, including against strong multilingual models.

  • Takeaways & Limitations

    Web-crawled data with high variability is preferable to Wikipedia-based data, and strong Transformer models can be trained with far fewer resources than English-scale pretraining.

  • Takeaways & Limitations

    With a BASE architecture and 4GB of training data, validation loss was still decreasing beyond 100k steps, suggesting the dataset was under-fit.

Abstract

from arXiv · show

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

1 Introduction

The paper addresses the limited availability of high-performing monolingual pretrained language models outside English by training and evaluating CamemBERT for French. It reports strong results across four tasks and finds that small, diverse corpora can rival much larger datasets.

  • Motivation: Large-scale pretrained language models have been less practically available for non-English languages because pretraining can require hundreds of gigabytes of text and hundreds of GPUs.Multilingual models can also lag behind monolingual counterparts for high-resource languages such as French.
  • Approach: CamemBERT is a French monolingual language model based on RoBERTa, trained using OSCAR corpora and evaluated on four downstream tasks.The tasks are POS tagging, dependency parsing, NER and NLI.
  • Results: CamemBERT improves on the state of the art in POS tagging, dependency parsing, NER and NLI against prior monolingual and multilingual approaches.The comparisons include mBERT, XLM and XLM-R.
  • Release: The work releases a French monolingual RoBERTa model trained on large-scale open-source OSCAR corpora.The model is publicly available under an MIT open-source license.
  • Corpus analysis: The study shows that small and diverse training sets can achieve performance similar to large-scale corpora by varying corpus size and domain.This analysis focuses on how pretraining-corpus characteristics affect downstream performance.

2 Previous work

Previous work progressed from contextual representations and multilingual pretrained models toward broader language coverage, but large-scale monolingual models remained concentrated in English. The paper positions CamemBERT as an effort to extend this paradigm to French.

  • Contextual representations: Pretrained representations evolved from non-contextual word embeddings to contextualized representations and large pretrained language models.Earlier representations were commonly used as inputs to task-specific architectures, whereas newer models can replace those architectures through fine-tuning.
  • Multilingual models: Multilingual BERT, XLM and XLM-R extended pretrained language modeling beyond English, with XLM-R trained on 2.5TB of data.mBERT was trained on Wikipedia data for 104 languages.
  • Monolingual models: Several non-English monolingual models had been released, including ELMo models for Japanese, Portuguese, German and Basque and BERT models for Chinese and German.These examples did not establish broad large-scale coverage across non-English languages.
  • Research gap: Before this work, no particular effort had trained models for languages other than English at a scale similar to the latest English models.The comparison point is RoBERTa, trained on more than 100GB of data.
  • Model architecture: RoBERTa improves BERT through dynamic masking, removal of next sentence prediction, larger batches, more data and longer training.BERT uses a bidirectional Transformer encoder trained with a masked language modeling objective.

3 Downstream evaluation tasks

The evaluation covers French POS tagging, dependency parsing, NER and NLI using established datasets and comparisons with multilingual, multitask and task-specific baselines. The datasets differ in annotation resources, task formulation and access conditions.

  • Tasks: CamemBERT is evaluated on four French tasks: POS tagging, dependency parsing, NER and NLI.POS tagging assigns grammatical categories, dependency parsing predicts labeled syntactic trees, NER identifies entity mentions, and NLI classifies sentence relations.
  • Syntactic evaluation: POS tagging and dependency parsing use the Universal Dependencies framework and four French UD v2.2 treebanks: GSD, Sequoia, Spoken and ParTUT.The Spoken treebank contains speech transcripts with annotated disfluencies and no punctuation.
  • Named entity recognition: NER uses the 2008 French Treebank with annotations containing more than 11 thousand entity mentions across seven entity types.The task predicts which words refer to people, locations, artifacts and organizations.
  • Data access: The French Treebank was stored and used only on Inria servers under a research-only agreement.This condition limits the dataset’s availability for the evaluation setup.
  • Natural language inference: NLI uses the French XNLI dataset, whose examples classify whether a hypothesis is entailed, neutral or contradictory with respect to a premise.XNLI provides 122k training, 2490 development and 5010 test examples per language, with accuracy as the metric.
  • Baselines: Baselines include mBERT and XLM for several tasks, UDify and UDPipe Future for syntactic tasks, CRF and BiLSTM-CRF for NER, and XLM-R for NLI.The study also compares against UDPipe Future enhanced with mBERT and Flair.

4 CamemBERT: a French Language Model

CamemBERT is a French monolingual RoBERTa-based language model trained on OSCAR data with SentencePiece tokenization and masked language modeling. It is used either by fine-tuning task-specific heads or by extracting frozen contextual embeddings for downstream tasks.

  • Training data: CamemBERT uses 138GB of French OSCAR text, containing 32.7B subword-tokenized tokens, as its pretraining corpus.OSCAR is extracted from Common Crawl and filtered with a language-classification model.
  • Pre-processing: A 32k-token SentencePiece vocabulary segments text into subwords without requiring language-specific pre-tokenization.SentencePiece extends BPE and WordPiece while avoiding word-level pre-tokenization.
  • Language modeling: CamemBERT follows BERT and RoBERTa as a bidirectional Transformer and uses BASE and LARGE architectures with whole-word masking.Its main architectural differences from RoBERTa are whole-word masking and SentencePiece tokenization.
  • Language modeling: The model is trained with masked language modeling: 15% of selected tokens are replaced, unchanged, or randomized before predicting the originals.The replacement proportions are 80% masked, 10% unchanged, and 10% random.
  • Language modeling: Dynamic masking varies the prediction targets across training and removes the next-sentence prediction objective.The paper states that dynamic masking improves variability and robustness across multiple epochs.
  • Downstream use: CamemBERT supports both end-to-end fine-tuning with task-specific predictive layers and feature-based use through frozen contextual embeddings.Feature-based representations average subword vectors and the last four Transformer layers; they are evaluated for POS tagging, parsing, and NER.

5 Evaluation of CamemBERT

CamemBERT is evaluated on French POS tagging, dependency parsing, NER, and NLI using both fine-tuning and contextual-embedding settings. It reaches state-of-the-art or improved performance across the four tasks, while often matching or exceeding more complex or multilingual alternatives.

  • POS tagging and dependency parsing: CamemBERT reaches state-of-the-art scores on all treebanks and metrics for POS tagging and dependency parsing in both evaluation settings.Fine-tuning has a slight advantage over feature-based embeddings, questioning the need for complex task-specific architectures.
  • POS tagging and dependency parsing: +4.15% LAS on Sequoia and +5.37% LAS on ParTUT are CamemBERT’s reported gains over UDify.Fine-tuning achieves these gains without a task-specific architecture and with a simpler optimization process.
  • POS tagging and dependency parsing: CamemBERT is slightly better overall than UDPipe Future+mBERT+Flair, but trails it by 0.03% and 0.14% UPOS on two treebanks.The exceptions are POS tagging on Sequoia and Spoken, respectively.
  • Named-entity recognition: CamemBERT achieves higher F1 scores than traditional CRF architectures and fine-tuned multilingual BERT models for NER.This holds both when fine-tuning CamemBERT and when using it as input embeddings for an LSTM+CRF architecture.
  • Named-entity recognition: 89.55 versus 89.08 shows slightly higher NER performance for CamemBERT embeddings with LSTM+CRF than for fine-tuning CamemBERT.The result indicates that the contextual embeddings remain useful with a task-specific downstream architecture.
  • Natural language inference: CamemBERTLARGE reaches 85.7% accuracy on French XNLI, compared with 85.2% for XLM-RLARGE.The BASE model also exceeds its multilingual counterparts, including gains of +5.6% over mBERT and +2.3 over XLMMLM-TLM.
  • Overall results: CamemBERT improves the state of the art across all four downstream tasks and performs well both when fine-tuned and when used as contextual embeddings.The paper presents this as confirmation of Transformer-based pretrained models’ usefulness for French.

6 Impact of corpus origin and size

The experiments compare French pretraining corpora by origin, homogeneity, and size. Heterogeneous web-crawled data outperforms Wikipedia, while 4GB of OSCAR performs similarly to 138GB on most tasks.

  • Corpus origin: 4GB samples of OSCAR and CCNet consistently outperform 4GB of French Wikipedia in both fine-tuning and embedding settings.The gap is larger for tagging and parsing on the Spoken treebank and for XNLI.
  • Corpus origin: The 4GB OSCAR and CCNet models have much more similar downstream performance.Both are Common-Crawl-based corpora, despite different filtering processes.
  • Data size: The 4GB OSCAR model performs similarly to the standard model trained on 138GB of OSCAR, except for a 0.9-point NER F1 gap favoring the larger model.The authors suggest that the larger corpus contains more named entities, while other tasks show little benefit from additional data.
  • Data size: 4GB of heterogeneous OSCAR or CCNet text is sufficient to reach state-of-the-art results with the BASE architecture, exceeding mBERT trained on 60GB.The authors note that this may reduce training cost, but the conclusion may not extend to larger architectures without further research.
  • Data size: The 4GB OSCAR model slightly exceeds the 138GB model on XNLI accuracy, 81.88 versus 81.55, possibly because each model was pretrained with a single random seed.This difference is presented as potentially seed-dependent rather than as a robust corpus-size effect.

7 Discussion

Subsequent work extended the monolingual French-language-model landscape and showed that combining FlauBERT with CamemBERT can improve parsing. CamemBERT-based systems also supported legal-text anonymization and French question answering.

  • Later monolingual models: Many monolingual language models appeared after CamemBERT, covering as many as 30 languages and generally outperforming multilingual models in tested configurations.The discussion cites FlauBERT and other later models as examples.
  • Model complementarity: An ensemble combining FlauBERT and CamemBERT improved French constituency parsing and established a new state of the art.The discussion highlights the complementarity of the two French RoBERTa-based models.
  • Applications: CamemBERT-based models enabled large-scale legal-text anonymization and early large French question-answering experiments.These applications followed the availability of comparable-scale French language models.

8 Conclusion

The paper demonstrates that a French RoBERTa-based model can match or improve state of the art across four tasks, including against strong multilingual models. It further finds that variable web data and small pretraining corpora can suffice, while some questions remain open.

  • Contribution: CamemBERT is a French RoBERTa-based language model evaluated on POS tagging, dependency parsing, NER, and NLI.The model was trained to assess monolingual Transformer pretraining for languages other than English.
  • Results: CamemBERT reaches or improves the state of the art on all four tasks, including against mBERT, XLM, and XLM-R, with fewer parameters.This is the paper’s main cross-task result.
  • Pretraining data: High-variability web-crawled data is preferable to Wikipedia-based data for the reported experiments.The conclusion states this as an experimental finding about pretraining-data origin.
  • Pretraining data: Models achieve high performance with as little as 4GB of pretraining data, questioning the need for very large pretraining corpora.The conclusion scopes this finding to the experiments reported in the paper.
  • Open question: Whether small domain-specific pretraining is preferable to transfer learning remains an open question for future work.The conclusion explicitly leaves this comparison unresolved.
  • Availability: CamemBERT is freely available under the MIT license through fairseq, Hugging Face, and the project website.The model uses pure open-source corpora.

A Impact of Whole-Word Masking

Whole-word masking provides a small NLI benefit but does not improve the lower-level tagging and parsing tasks. Larger architectures help NLI and NER, whereas POS tagging and dependency parsing show no significant difference from BASE.

  • Whole-word masking: Whole-word masking improves NLI performance by 0.5 accuracy points.The reported benefit is specific to natural language inference.
  • Whole-word masking: Whole-word masking does not benefit NER, POS tagging, or dependency parsing.The authors describe this lack of improvement as unexpected for these lower-level tasks.
  • Model size: The LARGE architecture yields 19.7% and 23.7% error reductions on NLI and NER, respectively.These models were trained with the 135GB CCNet corpus.
  • Model size: Tripling the parameter count does not produce a significant difference from BASE on POS tagging and dependency parsing.The comparison concerns the reported Table 8 results.

C Impact of training dataset

The comparison examines CamemBERT models trained on CCNet and OSCAR, whose filtering differences affect downstream performance differently across tasks.

  • OSCAR-pretrained CamemBERT performs slightly better than the CCNet model on POS tagging and dependency parsing.The datasets mainly differ because CCNet applies an additional filter favoring Wikipedia-like texts.
  • +1.31 improvement on NER is reported for the OSCAR model relative to the CCNet model.
  • +0.67 NLI performance is reported for the CCNet model relative to the OSCAR model.

D Impact of number of steps

The study tracks downstream performance as pretraining steps increase, finding that simpler tasks plateau earlier while NLI continues improving more gradually.

  • 22k steps marks an early performance plateau for dependency parsing and NER.Scores are averaged across at least four runs and, for these tasks, across four treebanks.
  • 100k steps still yields slowly increasing NLI performance despite smaller marginal gains with additional pretraining.
  • 500k steps produces little additional POS-tagging or parsing improvement over 100k steps, but increases XNLI by +0.84.
  • Low-level syntactic representations are captured earlier than the complex semantic information required for NLI.
Loading 1911.03894v3…