Source-linked AI summary
IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization
Fajri Koto, Jey Han Lau, Timothy Baldwin
TL;DR
The paper addresses adapting pretrained Indonesian BERT to Indonesian Twitter when domain-specific words do not match the existing vocabulary. It extends INDOBERT with Twitter vocabulary and compares initialization methods for new embeddings. Averaged BERT subword initialization achieves the best average performance across seven tasks and makes pretraining five times faster than training from scratch.
Problem
General pretrained models can struggle with domain-specific text, and Indonesian Twitter is underrepresented despite its broad practical use.
Method
IndoBERTweet adapts monolingual Indonesian BERT to Twitter by adding domain-specific vocabulary and comparing initialization strategies for new word types.
Results
The averaged subword-embedding method achieves the best average performance across seven tasks and is five times faster than pretraining from scratch.
Takeaways & Limitations
Average subword initialization provides an effective and efficient approach for adapting Indonesian BERT to Twitter vocabulary.
Takeaways & Limitations
The English comparison is not direct because BERTWEET and the adapted ROBERTA use different tokenization methods and vocabularies.
Abstract
from arXiv · showhide
We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.
1 Introduction
Domain-specific text remains difficult for general pretrained language models, especially when domain words are poorly represented by the existing vocabulary. IndoBERTweet addresses Indonesian Twitter by adapting Indonesian BERT and comparing vocabulary-initialization strategies.
- Domain-specific data such as social media remains challenging for pretrained language models despite their broad NLP success.
- Poor tokenization of domain words is a central drawback of domain-adaptive pretraining.In BioBERT, “Immunoglobulin” is split into seven subword tokens despite being common in biology.
- Prior vocabulary-extension methods use word2vec projections or random initialization with weight augmentation.
- IndoBERTweet extends monolingual Indonesian BERT for Twitter and evaluates domain-specific vocabulary initialization strategies.
- Indonesian is underrepresented in NLP, while Twitter supports policy, business, election, and health applications.
- The model is presented as the first large-scale pretrained Indonesian social-media model, while average subword initialization reduces domain-adaptive pretraining overhead by 80%.
2 INDOBERTWEET
IndoBERTweet is trained on a large, filtered Indonesian Twitter corpus and adapts INDOBERT by replacing its vocabulary with Twitter-specific types. New embeddings are compared using four initialization strategies, including averaged INDOBERT subword embeddings.
- The corpus covers economy, health, education, and government using 60 keywords collected from December 2019 to December 2020.
- 26M tweets containing 409M word tokens were collected, with 230K tweets reserved for development.
- IndoBERTweet follows INDOBERT’s masked-language-model training procedure with a 12-layer transformer and maximum sequence length of 128 tokens.
- Five models are trained: one from scratch and four using domain-adaptive pretraining with different vocabulary adaptation strategies.
- INDOBERT’s 31,923-type vocabulary is replaced by a 31,984-type Twitter vocabulary containing 14,584 new types and 17,400 shared types.
- New word types use uniform random, Gaussian random, fastText linear projection, or averaged INDOBERT subword initialization.
- For a word x, averaged initialization computes the mean INDOBERT embedding over the WordPiece tokens produced for x.
3 Experimental Setup
The experiments compare training and vocabulary-adaptation settings, then evaluate the resulting models across seven Indonesian Twitter datasets and several established baselines.
- Pretraining from scratch uses 1M steps, a 1e−4 learning rate, Adam, and four V100 GPUs.
- Domain-adaptive experiments compare no vocabulary adaptation, vocabulary adaptation without further pretraining, and adaptation with 200K additional steps.
- Evaluation covers seven Indonesian Twitter datasets spanning sentiment, emotion, hate speech, and named entity recognition.
- Classification uses an MLP over average-pooled outputs, whereas NER predicts tags from each word’s first subword.
- Baselines include two INDOBERT models, multilingual BERT, and MalayBERT.
4 Experimental Results
Across seven Indonesian Twitter datasets, domain-specific pretraining and vocabulary adaptation improve performance over off-the-shelf models, with average subword initialization enabling especially efficient adaptation.
- Table 2 compares pretrained models with different adaptive pretraining strategies across seven Indonesian Twitter datasets.
- INDOBERTWEET trained from scratch for 1M steps improves average performance by almost +3% absolute.
- Vocabulary adaptation without pretraining reduces performance overall, with Average of Subwords producing the smallest gap at −0.2% absolute.
- After 200K target-domain pretraining steps, all vocabulary-initialization methods improve substantially.
- The embedding averaging method achieves the best results after adaptation, outperforming INDOBERTWEET trained from scratch for 1M steps.
5 Discussion
The English comparison evaluates embedding-averaging adaptation against BERTWEET while highlighting limitations from smaller pretraining data and mismatched tokenization. Indonesian and English models also differ in how newly added types are split into subwords.
- The English comparison is not direct because BERTWEET uses byte-level BPE whereas ROBERTA uses fastBPE.
- The English domain-adaptive pretraining uses 42M tweets, an order of magnitude fewer than BERTWEET’s 845M tweets, and runs for 200K steps.
- BERTWEET outperforms ROBERTA by +3.4% absolute, while domain-adaptive pretraining narrows the gap to +2.2%.
- BERTWEET and INDOBERTWEET differ in casing, tokenizer, and vocabulary size, which may contribute to their different results.
- New types average 2.6 subwords in INDOBERTWEET and 3.4 in BERTWEET after tokenization.
6 Conclusion
The paper introduces a large-scale pretrained model for Indonesian Twitter and evaluates vocabulary adaptation strategies for domain-adaptive pretraining. Average-pooling original-model subword embeddings gives the best seven-task average and is five times faster than pretraining from scratch.
- The paper presents the first large-scale pretrained model for Indonesian Twitter.
- Average-pooling subword embeddings achieves the best average performance across 7 tasks among the evaluated vocabulary-adaptation strategies.
- Average-pooling subword embeddings is five times faster than pretraining from scratch.
A Results with English BERTWEET
Table 4 reports English test-set results using task-specific metrics and data, metrics, and splits based on Nguyen et al. (2020).
- Table 4 reports English results over test sets using accuracy, AvgRec, F1pos, and F1entity for the listed task groups.
- The experiments were rerun, with slightly lower performance for some models compared with BERTWEET.