Source-linked AI summary
RobBERT: a Dutch RoBERTa-based Language Model
Pieter Delobelle, Thomas Winters, Bettina Berendt
TL;DR
Multilingual models often underperform monolingual models on language-specific tasks, motivating a Dutch pre-trained model. The paper trains and evaluates RobBERT, a RoBERTa-based Dutch model, finding stronger results across tasks, especially with smaller datasets, while identifying corpus-order and translation-sampling limitations.
Problem
Multilingual BERT can perform well, but models trained on a single language have been shown to outperform multilingual versions for that language.
Method
The paper pre-trains RobBERT with the RoBERTa regime on Dutch data and evaluates task performance, dataset-size effects, tokenizers, and fairness.
Results
RobBERT outperforms earlier and other BERT-based approaches across several Dutch tasks, with significant advantages on smaller datasets.
Takeaways & Limitations
RobBERT is presented as a useful base model for fine-tuning across a large range of Dutch language application domains.
Takeaways & Limitations
RobBERT’s shuffled OSCAR corpus does not place subsequent lines together, limiting learning of long-range dependencies compared with RoBERTa’s sentence arrangement.
Abstract
from arXiv · showhide
Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multilingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model's fairness. We found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large variety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP applications.
1 Introduction
Dutch-specific language models may outperform multilingual models on Dutch NLP tasks. The paper introduces RobBERT, a Dutch RoBERTa-based model, and evaluates its tasks, tokenizer, dataset-size sensitivity, and fairness.
- Pre-training on large unlabeled corpora followed by fine-tuning on smaller annotated datasets supports many NLP tasks.
- Monolingual models usually improve multilingual models on tasks in their specific language.
- RobBERT is introduced as a Dutch RoBERTa-based pre-trained language model.
- The paper evaluates RobBERT on new zero-shot tasks, smaller datasets, language-specific tokenization, and fairness.
2 Related Work
Related work established transformers, BERT, and RoBERTa as powerful contextual language-model architectures. It also motivated examining multilingual models, coreference-related capabilities, and fairness risks in language-model predictions.
- BERT uses bidirectional context, while transformer models generate contextualized word embeddings and support coreference resolution.
- RoBERTa critically evaluated and improved BERT’s training approach, particularly its use of the NSP task.
- Multilingual BERT is trained on Wikipedia corpora in 104 languages, while monolingual models such as CamemBERT and BERT-NL outperformed mBERT on reported tasks.
3 Pre-training RobBERT
RobBERT was pre-trained with the RoBERTa regime on a large Dutch OSCAR corpus, using versions that isolate the effect of a Dutch tokenizer. Its architecture follows RoBERTa base and uses masked language modeling without NSP.
- RobBERT v1 replaces RoBERTa’s pre-training corpus with Dutch data, while RobBERT v2 also replaces its tokenizer with a Dutch version.
- The Dutch OSCAR corpus contains 39GB, 6.6 billion words, and 126 million lines of text.
- RobBERT v2 uses a Dutch byte-level BPE tokenizer with a vocabulary limited to 40k words.
- RobBERT uses RoBERTa base’s 12 self-attention layers, 12 heads, and 117M trainable parameters.
- Pre-training uses masked language modeling without the next-sentence prediction task.
4 Evaluation
RobBERT was evaluated across Dutch classification, zero-shot, POS-tagging, and named-entity-recognition tasks, including settings with limited training data. Results generally favored RobBERT, especially on smaller datasets, although multilingual BERT remained competitive on POS tagging and a prior multilingual result led NER.
- Evaluation scope: RobBERT was evaluated on sentiment analysis, die/dat disambiguation, POS tagging, and named entity recognition, including full-data, reduced-data, and zero-shot settings.The evaluation used Dutch Book Reviews, Europarl die/dat examples, Universal Dependencies Lassy, and CoNLL-2002.
- Die/Dat Disambiguation: RobBERT outperforms previous models and other BERT models both with and without fine-tuning on die/dat disambiguation.The comparison includes zero-shot masked-word prediction and fine-tuned classification approaches.
- Die/Dat Disambiguation: A Dutch tokenizer halves the error rate for die/dat disambiguation when the model is fine-tuned on the full dataset.The paper attributes the task advantage of BERT-based models over the prior RNN approach partly to better coreference resolution.
- Part-of-speech Tagging: Multilingual BERT marginally outperforms both Dutch BERT models on POS tagging, but the difference is not statistically significant.The authors suggest transferable language structures or limitations of the UD Lassy dataset as possible explanations.
- Part-of-speech Tagging: RobBERT v2 outperforms all other models when POS tagging uses 1,000 data points or less, while models perform similarly on the full UD Lassy dataset.The POS evaluation uses 17 Universal Dependencies tags and reports accuracy with confidence intervals.
- Named Entity Recognition: Wu and Dredze’s multilingual BERT result leads the NER comparison with an F1 score of 90.94, while the authors’ token-labeling multilingual BERT reaches 84.19.The reported NER scores use the CoNLL-2002 evaluation setting, with some results drawn from prior work.
5 RobBERT and Fairness
RobBERT’s fairness evaluation examines gender stereotyping in professions and unequal predictive performance across reviewer genders. The results reveal persistent male-pronoun preference and metric-dependent fairness differences.
- 5.1 Gender Stereotyping: Template-based association tests evaluate whether RobBERT associates professions with gendered pronouns.The model’s masked-language-model head fills pronouns while a second template slot iterates over professions.
- 5.1 Gender Stereotyping: In the control template, RobBERT ranks the male pronoun as more likely for every instance despite no expected correlation with profession gender.The authors suspect greater coverage of men in the training corpus contributes to this effect.
- 5.1 Gender Stereotyping: RobBERT’s pronoun rankings correlate with professions’ associated gender when the pronoun and profession refer to the same entity.The model still ranks the male pronoun as more likely in almost all cases; “actrice” is the sole exception.
- 5.2 Unequal Predictive Performance: Fairness analysis of sentiment classification uses Demographic Parity Ratio and Equal Opportunity on reviews annotated with binary gender.The held-out DBRD test set was augmented using self-reported gender from author profiles, and 76% of gender-labelled reviews were written by women.
- 5.2 Unequal Predictive Performance: For positive reviews, the EO difference is 0.028 and DPR is 70.2%, but conditioning EO on the true outcome yields a difference close to 0.The DPR falls below the often-used 80% fairness threshold, while ROC curves help explain the EO result through predictive performance.
- 5.2 Unequal Predictive Performance: For highly positive reviews, fairness differences become more pronounced and the model has better predictive performance for reviews written by women.The evaluation uses ROC curves split by reviewer gender and rating level.
7 Limitations and Future Work
The paper identifies several directions for improving RobBERT-like models and extending RobBERT to new applications. These include training, corpus, tokenizer, fairness, and downstream-task extensions.
- Limitations and Future Work: Future RobBERT-like models could use newly developed unsupervised pre-training tasks, unshuffled OSCAR text, morpheme-based Dutch tokens, and fairness improvements during pre-training.The authors specifically note sentence order prediction, long-range dependencies, Dutch compounds, and downstream representational harm as relevant directions.
- Limitations and Future Work: RobBERT could support encoder–decoder models and many Dutch language tasks not examined in this paper.The authors expect its state-of-the-art performance on small and large datasets to help advance results when fine-tuned on new datasets.
8 Conclusion
RobBERT is a Dutch RoBERTa-based language model that outperforms earlier and other BERT-based approaches across several Dutch language tasks, especially with smaller datasets.
- RobBERT is a Dutch language model based on RoBERTa.
- RobBERT outperforms earlier approaches and other BERT-based language models across several Dutch language tasks.
- RobBERT significantly outperforms other BERT-like models when dealing with smaller datasets, making it useful across many application domains.