Source-linked AI summary
BERTje: A Dutch BERT Model
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, Malvina Nissim
TL;DR
BERTje addresses the limited availability of high-performing Dutch language models by training and evaluating a monolingual Dutch BERT model on diverse text. It consistently outperforms multilingual BERT on word-level Dutch NLP tasks, while sentence-level coherence tasks remain to be investigated.
Problem
Dutch NLP can use multilingual BERT, but its multilingual training and Wikipedia-only data may be less suited to Dutch general language use than a monolingual model.
Method
The authors train BERTje with BERT's architecture and parameters on a 2.4B-token Dutch pre-training dataset, then fine-tune it across several Dutch NLP tasks against multilingual BERT.
Results
BERTje consistently outperforms multilingual BERT on Dutch word-level NLP tasks, including part-of-speech tagging, named-entity recognition, semantic role labeling, and spatio-temporal relation evaluation.
Takeaways & Limitations
The results indicate that a monolingual Dutch model should be preferred over multilingual BERT for the evaluated Dutch word-level NLP tasks.
Takeaways & Limitations
Performance may be affected by overfitting, and BERTje's performance on sentence-level tasks requiring coherence between sentences remains to be investigated.
Abstract
from arXiv · showhide
The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.
1 Introduction
BERT shifted NLP toward fine-tuning general-purpose transformer language models, but its success was primarily concentrated in English. For Dutch, the paper develops BERTje to test whether a diverse monolingual model can outperform multilingual BERT.
- BERT and its derivatives improved state-of-the-art results on many benchmark NLP tasks through fine-tuning.
- Multilingual BERT covers Dutch through Wikipedia text spanning 104 languages, but Wikipedia is not representative of general language use.
- Monolingual models may perform better on tasks in a specific language than multilingual BERT.
- BERTje is a Dutch BERT model developed from multi-genre data and evaluated by fine-tuning it on several Dutch NLP tasks.
- The paper compares BERTje's performance across all evaluated tasks with multilingual BERT.
2 Pre-training data and parameters
BERTje uses a BERTBASE-equivalent architecture with Dutch pre-training data assembled from multiple genres. Its 2.4B-token corpus and modified pre-training objectives are designed for broad Dutch language coverage.
- BERTje uses a BERTBASE-equivalent architecture with 12 transformer blocks and a dataset intended to match English BERT in size and diversity.
- The pre-training corpus combines books, Dutch news, a multi-genre reference corpus, web news, and Wikipedia.
- 12GB of cleaned text produced the final pre-training dataset, containing about 2.4B tokens after overlap and quality filtering.
- BERTje uses a 30K-token WordPiece vocabulary created from the raw Dutch pre-training dataset.
- Because NSP was ineffective, BERTje uses sentence-order prediction, pairing each sentence with either the next or previous sentence.
- The modified MLM strategy masks consecutive WordPiece tokens belonging to the same word, selecting 15% of all tokens.
- BERTje was pre-trained for 1M iterations, with downstream performance also evaluated at an 850K checkpoint.
3 Tasks and test data
The evaluation covers Dutch named-entity recognition, part-of-speech tagging, several extracted linguistic classification tasks, and binary sentiment analysis. These tasks use established annotated corpora and document-level splits where specified.
- The evaluation includes CoNLL-2002 four-class BIO-encoded named-entity recognition and 16 universal POS tags from the Lassy Small treebank.
- SoNaR-1 supplies token-level NER, coarse POS, and fine-grained POS classification with 241 total fine-grained labels.
- SoNaR-1 annotations also cover semantic-role predicates and arguments, semantic-role modifiers, and spatio-temporal relations.
- The extracted SoNaR-1 annotations are flattened to token-level classifications and split into 80% training, 10% validation, and 10% test documents without overlap.
- Sentiment analysis uses the balanced 110k Dutch Book Reviews Dataset for binary positive-versus-negative classification.
4 Results
BERTje is evaluated against multilingual BERT across Dutch NLP tasks using consistent fine-tuning settings. It outperforms multilingual BERT on named-entity recognition, part-of-speech tagging, semantic role labeling, and spatio-temporal relation data, while also approaching a tuned sentiment-analysis benchmark.
- Named-Entity Recognition: BERTje outperforms multilingual BERT on named-entity recognition for both CoNLL-2002 and SoNaR-1 data.Performance also improves after the 850K checkpoint, although it does not exceed the reported 90.9% CoNLL-2002 state-of-the-art test score.
- Part-of-Speech tagging: BERTje consistently outperforms multilingual BERT on part-of-speech tagging across all three tag sets.The 850K and fully trained BERTje models differ by at most 0.3 percentage points, indicating that relevant tagging information was learned before 850K iterations.
- Semantic Roles and Spatio-Temporal Relations: BERTje outperforms multilingual BERT on both semantic role labeling and spatio-temporal relation test data.These tasks show little improvement after the 850K checkpoint, and the reported results are stand-alone because comparable systems were not identified.
- Results: Training, development, and test scores show a pattern consistent with possible overfitting, suggesting task-specific hyperparameter tuning may improve performance.The comparison uses results from Tables 1 and 3.
5 Conclusion
BERTje consistently outperforms multilingual BERT on word-level Dutch NLP tasks, supporting the use of a monolingual model. Pre-training duration appears to matter more for higher-level linguistic information, while sentence-level coherence remains to be investigated.
- BERTje consistently outperforms multilingual BERT on word-level NLP tasks, indicating that a monolingual model should be preferred for Dutch.
- Lower-level linguistic structures, such as part-of-speech tags, appear to be learned earlier during pre-training than higher-level information.
- Low-level linguistic tasks do not benefit from pre-training beyond 850K iterations, whereas higher-level entity recognition does benefit from longer training.
- The results indicate that higher-level linguistic structures may be properly learned only after lower-level structures have been encoded.
- Future work should examine BERTje’s layer-wise linguistic abstractions and its performance on sentence-level tasks requiring coherence between sentences.