Source-linked AI summary
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
Fajri Koto, Afshin Rahimi, Jey Han Lau, Timothy Baldwin
TL;DR
Indonesian NLP lacks comprehensive datasets, language resources, and standardized evaluation despite the language’s large speaker population. The paper releases INDOLEM, a seven-task benchmark, and INDOBERT, then evaluates the model across the benchmark. INDOBERT achieves state-of-the-art performance over the dataset and outperforms other methods on the reported semantic tasks.
Problem
Indonesian NLP is under-represented because annotated datasets, language resources, and resource standardization are limited.
Method
The paper releases INDOLEM, a standardized seven-task Indonesian benchmark, and develops the monolingual pre-trained language model INDOBERT.
Results
INDOBERT achieves state-of-the-art performance over INDOLEM, outperforming all other methods on sentiment analysis and extractive summarization.
Takeaways & Limitations
INDOLEM provides a comprehensive benchmark for Indonesian NLP, while INDOBERT is shown to be superior to existing pre-trained models based on the benchmark.
Takeaways & Limitations
The study notes that a soon-to-be-released Liputan6 dataset would be substantially larger than the available summarization resources at the time.
Abstract
from arXiv · showhide
Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.
1 Introduction
Indonesian NLP is under-represented despite its large speaker population, reflecting limited datasets, language resources, and standardization. The paper addresses these gaps with INDOLEM and INDOBERT.
- Indonesian NLP is under-represented despite over 200M first-language speakers, amid shortages of annotated datasets, language resources, and resource standardization.
- INDOLEM standardizes splits and evaluation metrics across seven NLP tasks and eight sub-datasets spanning morpho-syntax, semantics, and discourse.Five sub-datasets build on previous work, while three are novel.
- INDOBERT is a monolingual pre-trained BERT language model developed for Indonesian following field best practices.
- Two contemporaneous Indonesian models also used the name IndoBERT, so the authors were not first to develop or name such a model.
- The paper releases INDOLEM as a comprehensive benchmark, introduces two novel discourse datasets, and releases INDOBERT for general use.The resources and associated code are publicly accessible through the project website.
2 Related Work
Prior work provides broad English, cross-lingual, and Mandarin benchmarks, but Indonesian lacks a comprehensive dataset for evaluating NLU systems. INDOLEM is introduced to fill this gap.
- GLUE and SuperGLUE benchmark English language understanding across diverse tasks, with SuperGLUE also including low-resource settings.SuperGLUE comprises eight tasks and was described as more challenging than GLUE.
- XGLUE covers nearly 20 languages and includes language generation tasks, while Universal Dependencies provides consistent annotations across 150 treebanks in 90 languages.
- Indonesian has no comprehensive dataset for evaluating NLU systems, unlike Mandarin Chinese and Vietnamese resources covering multiple tasks.INDOLEM is presented as the effort to fill this Indonesian benchmarking void.
3 INDOBERT
INDOBERT is a BERT-style transformer trained as a masked language model on Indonesian text. Its architecture and training corpus are described alongside the dataset resources summarized in Table 1.
- INDOBERT uses a BERT-style transformer and is trained purely as a masked language model with the Hugging Face framework.The configuration follows the default uncased BERT-Base setup.
- Table 1 summarizes the datasets incorporated in INDOBERT and marks datasets with canonical splits using an asterisk.
- INDOBERT has 12 hidden layers, 768-dimensional hidden states, 12 attention heads, 3,072-dimensional feed-forward layers, and a 31,923-token WordPiece vocabulary.
- The corpus produces 1,067,581 training instances and 13,985 development instances after preprocessing into 512-token document blocks.
4 INDOLEM: Tasks
INDOLEM organizes Indonesian NLP evaluation across morpho-syntax, semantics, and discourse coherence, combining existing resources with novel tasks and standardized benchmarking. It includes datasets for POS tagging, NER, dependency parsing, sentiment analysis, summarization, next tweet prediction, and tweet ordering.
- INDOLEM groups seven Indonesian NLP tasks into morpho-syntax/sequence labelling, semantics, and discourse coherence.
- Morpho-syntax/Sequence Labelling: The dataset uses Indonesian resources for POS tagging, NER, and dependency parsing, including corrected Universal Dependencies data.
- Semantics: INDOLEM includes binary sentiment classification from Twitter and hotel reviews, with aspect-level hotel annotations converted to sentence labels by majority polarity.
- Semantics: Indonesian summarization research has largely used extractive methods on small datasets, so INDOLEM adopts the IndoSum document–summary corpus for extractive evaluation.
- Discourse Coherence: The authors introduce next tweet prediction and message ordering to assess discourse coherence in Indonesian Twitter threads.
- Discourse Coherence: Next tweet prediction matches a 2–4-tweet premise against four candidate continuations, including the original response and three randomly selected distractors.
5 Evaluation Methodology
The evaluation methodology selects task-appropriate datasets, splits, and metrics across morpho-syntax, semantics, and discourse coherence. Cross-validation is used where appropriate, while fixed or canonical splits preserve comparability and prevent artefactual overlap.
- Morpho-syntax/Sequence Labelling: POS tagging uses 5-fold cross-validation with standard tag accuracy, while NER uses entity-level F1 after IOB2 conversion and validation-set reservation.
- Morpho-syntax/Sequence Labelling: Dependency parsing uses a predefined test set for UD-Indo-GSD and fixed 5-fold cross-validation for UD-Indo-PUD.
- Semantics: Sentiment analysis uses stratified 5-fold cross-validation and F1, whereas summarization uses canonical splits and ROUGE F1 metrics R1, R2, and RL.
- Discourse Coherence: Next tweet prediction avoids 5-fold cross-validation to prevent distractor overlap across splits and is evaluated with accuracy against a 25% random baseline.
- Discourse Coherence: Tweet ordering is evaluated by averaging Spearman’s rank correlation between gold and predicted thread orders.
6 Comparative Evaluation
The comparative evaluation tests BiLSTM and fastText baselines, task-specific extractive summarization systems, and BERT-based models across INDOLEM tasks. It also compares IndoBERT with multilingual and Malay monolingual BERT resources.
- Baselines: Most experiments use a BiLSTM with 300d Indonesian fastText embeddings as the baseline.
- Baselines: Extractive summarization baselines include a hidden-Markov-model sentence tagger, a hierarchical LSTM, and an oracle that greedily maximizes ROUGE.
- Baselines: For next tweet prediction, the baseline encodes premise tweets and candidate tweets with BiLSTMs, classifying each premise–candidate pair and selecting the highest-probability tweet.
- Baselines: Tweet ordering uses a hierarchical BiLSTM that encodes individual tweets before learning inter-tweet rank positions.
- BERT Benchmarks: IndoBERT is benchmarked against multilingual BERT and MalayBERT, whose usefulness reflects multilingual coverage and Malay–Indonesian similarity, respectively.
- BERT Benchmarks: BERT-based evaluation applies task-specific fine-tuning for sequence labelling, sentiment, next tweet prediction, tweet ordering, dependency parsing, and extractive summarization.
7 Results
IndoBERT generally outperforms competing models across Indonesian semantic, dependency-parsing, and discourse-coherence tasks, while POS tagging remains similar across contextual models. Results also expose task-specific exceptions and substantial room for improvement.
- Morpho-syntax and sequence labelling: MALAYBERT and IndoBERT outperform MBERT on NER, while all three contextual models perform similarly on POS tagging.The contextual models perform well above the BiLSTM baseline on POS tagging.
- Dependency parsing: IndoBERT outperforms all methods on both dependency-parsing metrics over UD-Indo-GSD.
- Dependency parsing: MBERT outperforms all methods including IndoBERT over UD-Indo-PUD, although comparisons are affected by different test partitions and evaluation setups.The cited passage notes that some prior results use the original dataset version or 10-fold cross-validation.
- Semantic tasks: IndoBERT outperforms all other methods on Indonesian sentiment analysis and extractive summarization.
- Semantic tasks: +13.2 points over naive Bayes and +7.5 points over MBERT are reported for sentiment analysis.
- Discourse coherence: IndoBERT is substantially better than the other models at discourse coherence modelling, despite not using next sentence prediction during training.Human performance was 90% for next tweet prediction, while human tweet-ordering rank correlation was ρ = 0.61, slightly higher than IndoBERT.
- Overall assessment: Substantial room for improvement remains across all tasks except possibly POS tagging and next tweet prediction.The authors hope IndoLEM will serve as a benchmark for tracking progress in Indonesian NLP.
8 Conclusion
The paper introduces IndoLEM, a seven-task Indonesian benchmark, and IndoBERT, a monolingual Indonesian pre-trained model. IndoBERT achieves state-of-the-art performance over the dataset.
- IndoLEM is a comprehensive Indonesian dataset spanning seven tasks in morpho-syntax, semantics, and discourse coherence.
- IndoBERT is a new BERT-style monolingual pre-trained language model for Indonesian.
- IndoBERT achieves state-of-the-art performance over IndoLEM after comparative evaluation against baselines and competing BERT models.