Source-linked AI summary
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti
TL;DR
Indonesian NLP lacks the datasets and benchmarks available for high-resource languages. This paper introduces IndoNLU, Indo4B, and IndoBERT models, achieving strong benchmark performance across twelve diverse Indonesian NLU tasks.
Problem
Indonesian NLP faces resource scarcity because existing benchmarks and datasets primarily support high-resource languages such as English and Chinese.
Method
The paper builds IndoNLU with twelve diverse tasks, collects approximately four billion Indonesian words in Indo4B, and trains IndoBERT and IndoBERT-lite as baseline models.
Results
IndoBERT models outperform multilingual models on 8 out of 12 tasks, while pre-trained and contextualized models outperform scratch and word-embedding baselines.
Takeaways & Limitations
IndoNLU, its datasets, models, code, and leaderboard provide a reproducible benchmark for advancing and comparing Indonesian NLU systems.
Takeaways & Limitations
The preprocessing retains original word forms to preserve contextual information from Indonesian compounding, affixation, and reduplication.
Abstract
from arXiv · showhide
Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in the natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for the training, evaluating, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset Indo4B collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, and thus it enables everyone to benchmark their system performances.
1 Introduction
Indonesian NLP lacks organized, reproducible resources despite widespread internet use. IndoNLU addresses this gap with a diverse benchmark, standardized splits, evaluation support, and Indonesian pre-trained models.
- Indonesian is the fourth largest language used over the internet, with around 171 million users, yet Indonesian NLP research remains slow-moving.
- Existing Indonesian datasets are scattered, poorly documented, and accompanied by limited community engagement.
- IndoNLU is the first Indonesian natural language understanding benchmark, comprising twelve diverse tasks across inputs, objectives, domains, and language styles.
- The benchmark standardizes dataset splits, provides pre-training code samples and downstream evaluation, and plans a public leaderboard for transparency.
- Indo4B contains around four billion words from varied public sources, supporting IndoBERT and IndoBERT-lite baselines that outperform most existing pre-trained models.
2 Related Work
Prior multilingual NLP benchmarks and contextual language models established evaluation practices, but these resources primarily target well-resourced languages and major language-specific settings.
- GLUE is an English multi-task NLU benchmark covering nine tasks, including single-sentence, semantic similarity, and natural language inference tasks.
- SuperGLUE extends evaluation to question answering, natural language inference, co-reference resolution, and word sense disambiguation.
- CLUE is a Chinese NLU benchmark with eight diverse tasks and a test set probing a Chinese-specific linguistic phenomenon.
- Contextual pre-trained models progressed from ELMo to BERT and RoBERTa, with BERT using transformer multi-head attention to eliminate recurrences.
- Language-specific and multilingual models include Chinese BERT and RoBERTa, French CamemBERT and FLAUBert, Multilingual BERT, and XLM.
3 IndoNLU Benchmark
IndoNLU organizes Indonesian NLU evaluation around twelve tasks, diverse datasets, standardized baselines, and a large self-supervised corpus called Indo4B.
- Downstream Tasks: IndoNLU comprises twelve tasks grouped into single-sentence classification, single-sentence sequence tagging, sentence-pair classification, and sentence-pair sequence labeling.
- Evaluation: The benchmark uses standardized evaluation and newly created dataset splits to support reproducible comparison across models.
- Downstream Tasks: The benchmark covers diverse domains and styles through datasets drawn from social media, reviews, news, Wikipedia revisions, hotel text, and banking discussions.
- Downstream Tasks: Tasks include emotion, sentiment, aspect-based sentiment, textual entailment, part-of-speech tagging, span extraction, keyphrase extraction, and named entity recognition.
- Indo4B Dataset: Indo4B contains around four billion words and approximately 250 million sentences spanning formal and colloquial Indonesian from twelve datasets.
- Baseline Models: Baseline models range from scratch and word-embedding systems to contextualized language models, including fastText, IndoBERT, IndoBERT-lite, Multilingual BERT, XLM, and XLM-R.
4 IndoBERT
IndoBERT and IndoBERT-lite are Indonesian contextual models trained on Indo4B with preprocessing and two-phase pre-training choices tailored to Indonesian text and available computation.
- IndoBERT and IndoBERT-lite are Indonesian BERT- and ALBERT-based contextual models trained using the Indo4B dataset.
- Dataset preparation separates paragraphs or lines to preserve contiguous sequences and distinguish sentences for BERT sentence-pair inputs.
- The models retain original word forms to preserve contextual information from Indonesian compounding, affixation, and reduplication.
- Twitter usernames, hashtags, emails, and links are masked in Twitter UI data but discarded from the larger Twitter Crawl dataset.
- IndoBERT models use SentencePiece BPE vocabularies with sizes 30.522 for IndoBERT and 30.000 for IndoBERT-lite.
- All IndoBERT models train in two phases, first with maximum sequence length 128 and then with maximum sequence length 512.
5 Results and Analysis
IndoNLU evaluation shows that pre-trained contextual models substantially outperform scratch and word-embedding baselines, while monolingual and multilingual models trade strengths across task types. IndoBERT-lite offers similar performance with lower computational cost, and Indo4B-trained fastText outperforms the larger CC-ID alternative.
- 5.1 Benchmark Results: IndoBERTLARGE, XLM-RLARGE, and IndoBERTBASE achieve the top-three classification results, while XLM-RLARGE, IndoBERTLARGE, and XLM-RBASE lead sequence labeling.
- 5.1 Benchmark Results: Pre-trained models outperform scratch models, while contextualized models substantially outperform word-embedding-based models.These results support the effectiveness of pretraining and the superiority of contextualized embeddings over word embeddings.
- 5.2 Performance-Space Trade-off: IndoBERT-lite models achieve similar performance to IndoBERT models with many fewer parameters and slightly lower computation cost.IndoBERT-liteLARGE is on par with XLM-RBASE while using 16x fewer parameters.
- 5.3 Multilingual vs. Monolingual Models: Contextualized monolingual models outperform multilingual models on classification, whereas multilingual models tend to perform better on sequence labeling, especially NERGrit and FacQA.NERGrit and FacQA contain many entity names from other languages, especially English.
- 5.4 Effectiveness of Indo4B Dataset: fastText-indo4b consistently outperforms fastText-cc-id across classification and sequence labeling settings despite Indo4B being much smaller.The reported corpus sizes are approximately 23 GB for Indo4B and 180 GB uncompressed for CC-ID; fastText-indo4b scores higher on 10 of 12 tasks.
- 5.5 Effectiveness of IndoBERT and IndoBERT-lite: Increasing the maximum sequence length to 512 in phase two improves sequence-labeling performance and enables learning temporal information from text.
6 Conclusion
The paper presents IndoNLU as a twelve-task Indonesian NLU benchmark and introduces Indo4B-trained IndoBERT and IndoBERT-lite models. It releases models, data, code, and a leaderboard to support reproducibility, benchmarking, and community engagement.
- IndoNLU is the first Indonesian NLU benchmark and contains 12 tasks spanning different difficulty levels, domains, and styles.
- Indo4B is a large clean Indonesian dataset used to train the monolingual contextual models IndoBERT and IndoBERT-lite.
- The authors release pre-trained models, collected data, code, and a leaderboard to support reproducibility, community engagement, and benchmark transparency.
A Data Samples
The appendix provides examples for the twelve IndoNLU downstream tasks. These examples cover classification, sequence labeling, and question-answering tasks identified by their task names and table locations.
- Examples for SmSA, EmoT, KEPS, HoASA, and CASA are provided in Tables 7–11.
- Examples for WReTE, NERGrit, NERP, BaPOS, and POSP are provided in Tables 12–16.
- Examples for FacQA and TermA are provided in Tables 17 and 18.
B Indo4B Data Sources
This section identifies the source of each corpus used to build Indo4B and points to Table 19 for the corpus sources. The supplied passages also identify sample-data tables for SmSA and EmoT.
- The corpus sources used to build Indo4B are presented in Table 19.
- Table 7 presents sample data for the SmSA task.
- Table 8 presents sample data for the EmoT task.
C Pre-Training Hyperparameters
This section reports the hyperparameters used to train IndoBERT and IndoBERT-lite, with the configuration provided in Table 20.
- The section covers hyperparameters for IndoBERT and IndoBERT-lite pre-training.
- The hyperparameter configuration is provided in Table 20.
- Table 20 is labeled as the configuration table for the IndoBERT and IndoBERT-lite pre-trained models.
D Fine-Tuning Hyperparameters
This section reports the hyperparameters used for fine-tuning each baseline model, with the configuration provided in Table 21.
- The section covers fine-tuning hyperparameters for each baseline model.
- The hyperparameter configuration is provided in Table 21.
- Table 21 contains the fine-tuning configuration for the baseline models.
E Extensive Experiment Results on IndoNLU Benchmark
The IndoNLU experiments use batch size 16 for most datasets and batch size 8 for FacQA and POSP. The supplied passages include sample-data tables and the full experiment-results table.
- Batch size 16 is used for all datasets except FacQA and POSP, which use batch size 8.
- Table 22 reports the results of all experiments conducted in the IndoNLU benchmark.
- Sample-data tables are provided for KEPS, HoASA, CASA, WReTE, NERGrit, NERP, BaPOS, POSP, FacQA, and TermA.
- The sample SmSA data include service-related categories such as air conditioning, hot water, odor, cleanliness, linen, service, sunrise meal, television, and WiFi.