Source-linked AI summary
Multilingual is not enough: BERT for Finnish
Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, Sampo Pyysalo
TL;DR
Multilingual BERT’s capabilities for lower-resourced languages remained insufficiently understood. The paper evaluates M-BERT against a Finnish BERT model trained from scratch across Finnish NLP tasks and diagnostic classifications. FinBERT systematically outperforms M-BERT and prior methods on the benchmark tasks, while probing results provide only weak support for its superiority on some tasks.
Problem
Research had not thoroughly established the capabilities of multilingual BERT for lower-resourced languages.
Method
The paper compiles Finnish pretraining data, trains language-specific FinBERT models, and compares them with M-BERT across Finnish NLP and diagnostic classification tasks.
Results
FinBERT systematically outperforms M-BERT and previously proposed methods across the benchmark tasks, establishing new state-of-the-art results for Finnish POS tagging, NER, and dependency parsing.
Takeaways & Limitations
Language-specific BERT models can provide substantial advances for a lower-resourced language such as Finnish, with similar benefits potentially extending to other languages with sufficient pretraining data.
Takeaways & Limitations
Probing-task results offer only weak support for FinBERT’s superiority, leaving open whether task coverage or implementation and data limitations explain the difference.
Abstract
from arXiv · showhide
Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at https://turkunlp.org/finbert .
1 Introduction
Prior BERT research focused mainly on English, leaving the capabilities of multilingual models for lower-resourced languages insufficiently understood. This paper evaluates multilingual and Finnish-specific BERT models on Finnish NLP tasks.
- Most deep transfer-learning work has concentrated on English, while other languages have typically been addressed through multilingual models.
- M-BERT was trained on pooled data from 104 languages and can generalize across languages, but monolingual BERT models have often outperformed it where available.
- The study introduces a Finnish BERT model trained from scratch and compares it with M-BERT across POS tagging, NER, dependency parsing, and diagnostic classification.
- On most tasks, M-BERT does not advance prior state-of-the-art performance, whereas the custom Finnish model systematically outperforms M-BERT and previous methods on benchmark tasks.
2 Related Work
Contextual transfer-learning models evolved from static word embeddings to Transformer-based BERT, whose success motivated multilingual and language-specific variants. BERT training requires substantial computational resources.
- Modern transfer learning incorporates textual context, addressing the one-vector-per-word limitation of earlier word-embedding methods.
- ELMo introduced bidirectional LSTM context embeddings, ULMFit targeted domain adaptation, and BERT used the attention-based Transformer architecture.
- BERT achieved strong performance across many tasks, while subsequent models included multilingual, cross-lingual, and domain-specific variants.
- Training BERT is computationally intensive and requires substantial resources.
3 Pretraining
FinBERT pretraining used diverse Finnish text sources, extensive cleaning, language-specific vocabularies, and BERT-compatible pretraining procedures. The resulting models were evaluated through vocabulary analyses and downstream resources.
- 3.1 Pretraining Data: FinBERT pretraining data combined Finnish news, online discussion, internet-crawl material, and Finnish Wikipedia texts.
- 3.1 Pretraining Data: The corpus was sentence-split, tokenized, parsed, deduplicated, and filtered to remove low-quality, machine-translated, and generated texts.
- 3.2 Vocabulary generation: Finnish-specific cased and uncased vocabularies contained 50,000 BPE word pieces and were evaluated against existing vocabularies on Finnish and English Wikipedia.
- 3.2 Vocabulary generation: FinBERT vocabularies produced more compact Finnish tokenization than M-BERT despite being half the multilingual vocabulary size.
- 3.2 Vocabulary generation: Uncased vocabulary generation strips accents, making two Finnish vowel pairs ambiguous, although the step is required for BERT compatibility.
- 3.3 Pretraining: Pretraining reused BERT’s masked-language-model and next-sentence-prediction tasks, with whole-word masking and balanced examples across sources.
4 Evaluation
The evaluation compares Finnish-specific and multilingual BERT across downstream and diagnostic tasks. FinBERT consistently outperforms M-BERT on the main Finnish benchmarks, while probing tasks provide only weak support for its broader language-model advantage.
- Evaluation setup: Experiments evaluate M-BERT and FinBERT on Finnish downstream NLP and diagnostic tasks, with repeated fine-tuning runs and development-set hyperparameter selection.The downstream tasks include POS tagging, named entity recognition, dependency parsing, and text classification.
- Part-of-speech tagging: FinBERT cased improves POS tagging over prior state of the art by 0.4–1.7% points across three Finnish treebanks.Its FTB error rate is less than half that of the best CoNLL’18 result; uncased FinBERT remains within approximately 0.1% points of the cased model.
- Named entity recognition: FinBERT establishes new state-of-the-art Finnish NER results, whereas M-BERT beats prior work in-domain but falls below FiNER-tagger out-of-domain.The in-domain test set is technology news, while the out-of-domain test set uses Wikipedia documents.
- Dependency parsing: Udify with cased FinBERT exceeds previous Finnish parsing results by 2.3–3.6% LAS points across three treebanks, reducing errors by 24%–31% relatively.Udify initialized with M-BERT does not outperform the strongest baseline.
- Text classification: Text-classification results differ sharply by domain, with discussion-corpus error rates more than doubling relative to news and domain coverage offered as a partial explanation.Only FinBERT pretraining included informal Finnish from online discussions; all models included Wikipedia-like material.
- Text classification: Language-specific BERT models outperform multilingual models across all training-data sizes for both Finnish text-classification datasets.On online discussion data, FinBERT leads M-BERT by over 20% points with the smallest training set and retains a 5% point advantage at 100,000 examples.
- Probing tasks: Probing results provide only weak support for FinBERT’s superiority, and the authors leave open whether this reflects task coverage or implementation and data limitations.The best language-specific model leads on all probing tasks except TreeDepth, where M-BERT performs best.
5 Discussion
Language-specific FinBERT clearly outperforms multilingual BERT for Finnish across a range of tasks. The findings suggest that similar benefits for other languages may depend on available pretraining data and can be supported by broadly reusable methods and open tools.
- Language-specific FinBERT clearly outperforms multilingual BERT across a range of Finnish NLP tasks.
- The feasibility of high-quality deep transfer learning models likely hinges on the availability of pretraining data.
- Finnish has online unannotated resources comparable in size to those used for FinBERT, suggesting potential for dedicated models in many languages.
- FinBERT’s training-data collection and filtering methods have few language dependencies and could be applied across languages with available UD resources.
6 Conclusions
The work compares Finnish-specific FinBERT models with multilingual BERT across a broad range of NLP tasks. The multilingual models fall behind previously proposed methods for most tasks, while the study releases its models and resources openly.
- The study compiles and filters a large unannotated Finnish corpus, trains language-specific FinBERT models, and evaluates them against multilingual BERT.The evaluation covers a broad range of natural language processing tasks.
- The multilingual models fall behind previously proposed methods for most tasks, failing to deliver on the promises of deep transfer learning for lower-resourced languages.
- FinBERT models, tools, and related resources are available under open licenses.