Source-linked AI summary
Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
Yuri Kuratov, Mikhail Arkhipov
TL;DR
The paper addresses how to obtain language-specific performance while retaining the transfer benefits of multilingual pre-training. It adapts multilingual BERT into a Russian monolingual model through Russian vocabulary and embedding reconstruction, and reports improved performance across Russian tasks, faster convergence, and open-sourced models.
Problem
The paper examines whether multilingual knowledge can be transferred into a Russian monolingual model while addressing the performance gap between multilingual and language-specific models.
Method
The method initializes a Russian monolingual BERT model from multilingual BERT and replaces its vocabulary and embeddings with Russian-specific ones.
Results
The adapted monolingual model substantially outperforms the multilingual model across common Russian NLP tasks, with improvements reported for paraphrase detection, sentiment analysis, and question answering.
Takeaways & Limitations
Multilingual initialization improves the adaptation of a Russian monolingual model while reducing training time, and the resulting Russian models are open sourced.
Abstract
from arXiv · showhide
The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performance on a wide range of tasks including reading comprehension, natural language inference, and sentiment analysis. At the moment there are two alternative approaches to train such models: monolingual and multilingual. While language specific models show superior performance, multilingual models allow to perform a transfer from one language to another and solve tasks for different languages simultaneously. This work shows that transfer learning from a multilingual model to monolingual model results in significant growth of performance on such tasks as reading comprehension, paraphrase detection, and sentiment analysis. Furthermore, multilingual initialization of monolingual model substantially reduces training time. Pre-trained models for the Russian language are open sourced.
1 Introduction
BERT established strong unsupervised pre-training results, while multilingual models offered cross-language transfer alongside monolingual alternatives. This work investigates transferring multilingual knowledge into a Russian monolingual model and reports substantially better performance than the multilingual model.
- BERT uses Transformer architecture and masked language modelling on large unlabeled Wikipedia text, achieving state-of-the-art English NLP results.
- Monolingual models had previously been reported to outperform multilingual models, despite multilingual models supporting multiple languages.
- The paper investigates multilingual-to-monolingual transfer using Russian as the target language.
- The authors show that a monolingual model can be trained using multilingual initialization.
- The monolingual setting achieves substantially better performance than the multilingual model across common Russian NLP tasks.
2 Model Architecture
The experiments use BERT, a Transformer encoder built from self-attention and trained with masked language modelling and next sentence prediction. The selected configuration is BERTBASE with 12 Transformer layers and task-specific output layers.
- BERT is a Transformer encoder whose basic building block is self-attention.
- The model was pre-trained on masked language modelling and next sentence prediction tasks.
- The experiments use the 12-layer BERTBASE configuration with hidden size 768, feed-forward size 3072, and 12 attention heads.
- Task-specific layers were trained according to the original BERT procedure.
3 Language transfer
The adaptation transfers multilingual BERT knowledge into a Russian monolingual model by replacing its vocabulary and embeddings while retaining multilingual initialization for the remaining parameters. A Russian-specific vocabulary is trained on Wikipedia and news data, and new embeddings are assembled from shared and newly initialized tokens.
- Multilingual subword tokenization can produce longer sequences for one language, which is undesirable because Transformer computation scales quadratically with sequence length.
- The method initializes the Russian monolingual model from multilingual BERT to reuse knowledge captured during multilingual training.
- A new Russian subword vocabulary is trained on Russian Wikipedia and news data, producing longer Russian words and subwords than the multilingual vocabulary.
- Shared-token embeddings remain unchanged, while tokens outside the vocabulary intersection are replaced by Russian tokens.
- New tokens are initialized with the mean embedding of tokens in the vocabulary intersection.
- The reassembled model is trained on the same data used to build the monolingual vocabulary and released through DeepPavlov.
4 Tasks description
The evaluation covers Russian paraphrase identification, sentiment analysis, and question answering. The datasets span labeled sentence pairs, social-network posts, and questions whose answers are text spans in corresponding contexts.
- The evaluation includes paraphrase identification, sentiment analysis, and question answering.
- ParaPhraser contains 7227 training pairs and 1924 test pairs labeled as precise paraphrases, near paraphrases, or non-paraphrases.
- RuSentiment contains 30521 VKontakte posts, each labeled with one of five sentiment classes.
- RuSentiment is more challenging because its informal social-network language differs from the Wikipedia and news data used for training.
- SDSJ Task B contains about 50,000 training and development questions and contexts, with answers represented as spans from their contexts.
5 Results
RuBERT was evaluated against multilingual BERT on paraphrase identification, sentiment analysis, and question answering, with results averaged across five runs. The Russian-specific vocabulary shortened mean sequences, while multilingual initialization accelerated convergence and reduced training time.
- RuBERT and multilingual BERT were evaluated on paraphrase identification, sentiment analysis, and question answering, averaging results across 5 runs.
- 5.1 Vocabulary comparison: RuBERT reduced mean sequence length by 1.6 times in subtokens compared with multilingual BERT.Both models use about 120k subtokens, but RuBERT's vocabulary was built specifically for Russian.
- 5.2 Training dynamics: Multilingual initialization reached the same loss at about 250 thousand steps that random initialization reached at about 800 thousand steps.Training 250 thousand steps took about two days on Tesla P100 x 8, saving six days of computational time.
- 5.2 Training dynamics: Averaging embeddings for new vocabulary subtokens had a positive effect on convergence rate compared with random initialization.
6 Conclusion
The paper concludes that multilingual pre-training and subsequent language-specific training improve Russian NLP performance. It also releases Russian pre-trained models and reproduction code through DeepPavlov.
- Multilingual initialization followed by language-specific unsupervised training improved performance across several Russian NLP tasks.
- Russian pre-trained models and code to reproduce the results were open sourced as part of the DeepPavlov library.