Source-linked AI summary
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Leylia Khodra, Ayu Purwarianti, Pascale Fung
TL;DR
Low-resource Indonesian, Javanese, and Sundanese lack broad public NLG benchmarks despite their importance and limited data. IndoNLG introduces a six-task benchmark, a multilingual pretraining corpus, and two local-language generation models. IndoBART and IndoGPT achieve competitive performance with about one-fifth of mBART-LARGE’s parameters and faster inference.
Problem
Publicly available NLG benchmarks are scarce for low-resource languages, although Indonesian, Javanese, and Sundanese are widely spoken and important for NLG.
Method
IndoNLG combines six NLG tasks, Indo4B-Plus pretraining data, and the IndoBART and IndoGPT models for Indonesian, Javanese, and Sundanese.
Results
IndoBART and IndoGPT achieve competitive performance across the benchmark with about 20% of mBART-LARGE’s parameters and faster CPU and GPU inference.
Takeaways & Limitations
Pretraining on closely related local languages can support efficient generation for very low-resource Javanese and Sundanese.
Takeaways & Limitations
The fairness and bias analysis provides only a partial picture of biases in models trained on Indonesian web data.
Abstract
from arXiv · showhide
Natural language generation (NLG) benchmarks provide an important avenue to measure progress and develop better NLG systems. Unfortunately, the lack of publicly available NLG benchmarks for low-resource languages poses a challenging barrier for building NLG systems that work well for languages with limited amounts of data. Here we introduce IndoNLG, the first benchmark to measure natural language generation (NLG) progress in three low-resource -- yet widely spoken -- languages of Indonesia: Indonesian, Javanese, and Sundanese. Altogether, these languages are spoken by more than 100 million native speakers, and hence constitute an important use case of NLG systems today. Concretely, IndoNLG covers six tasks: summarization, question answering, chit-chat, and three different pairs of machine translation (MT) tasks. We collate a clean pretraining corpus of Indonesian, Sundanese, and Javanese datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and IndoGPT. We show that IndoBART and IndoGPT achieve competitive performance on all tasks -- despite using only one-fifth the parameters of a larger multilingual model, mBART-LARGE (Liu et al., 2020). This finding emphasizes the importance of pretraining on closely related, local languages to achieve more efficient learning and faster inference for very low-resource languages like Javanese and Sundanese.
1 Introduction
IndoNLG addresses the shortage of publicly available NLG benchmarks for low-resource languages by introducing resources and standardized evaluation for Indonesian, Javanese, and Sundanese. It combines a multilingual benchmark, pretraining data, and models designed for these widely spoken languages.
- The benchmark responds to the scarcity of publicly available NLG resources for low-resource languages beyond machine translation.
- IndoNLG introduces a benchmark covering Indonesian, Javanese, and Sundanese, languages spoken by more than 100 million native speakers combined.
- IndoNLG combines clean pretraining data, pretrained models, standardized evaluation data, and downstream NLG tasks.
- The benchmark includes summarization, question answering, dialogue, English–Indonesian translation, and new Sundanese–Indonesian and Javanese–Indonesian translation datasets.
- IndoBART and IndoGPT use pretraining on related local languages to achieve strong performance on Javanese and Sundanese with fewer parameters and smaller pretraining data than multilingual models.
2 Related Work
Prior work provides low-resource NLU benchmarks, translation datasets, and pretrained generation models, but IndoNLG targets broader NLG evaluation for Indonesian and related regional languages.
- Existing low-resource benchmarks largely emphasize NLU, while related resources include machine translation, Indonesian summarization, and Indonesian dialogue datasets.
- IndoNLG extends this landscape with a broader benchmark for generation in Indonesian, Javanese, and Sundanese.
- Recent pretrained generation models include GPT-2, GPT-3, BART, and T5, establishing pretraining followed by fine-tuning as a successful NLG paradigm.
3 IndoNLG Benchmark
IndoNLG assembles six NLG subtasks from existing and newly created datasets, alongside a rebalanced Indonesian, Sundanese, and Javanese pretraining corpus. The benchmark includes translation, summarization, question answering, and chit-chat settings.
- 3.1 Indo4B-Plus Pretraining Dataset: Indo4B-Plus combines Indonesian data from Indo4B with Sundanese and Javanese text from Wikipedia and CC-100.
- 3.1 Indo4B-Plus Pretraining Dataset: Because local-language data comprise about 1% of the corpus, the authors rebalance sampling and increase their share to about 3%.
- IndoNLG contains six subtasks spanning summarization, question answering, chit-chat, and three language-pair translation tasks.
- 3.2 IndoNLG Tasks: The translation tasks evaluate English–Indonesian, Sundanese–Indonesian, and Javanese–Indonesian in both directions using parallel corpora and standardized splits.
- 3.2 IndoNLG Tasks: The benchmark also uses Indonesian news summarization, TyDiQA Indonesian question answering, and XPersona Indonesian dialogue without persona information.
4 Experimental settings
The experiments compare pretrained local-language models with scratch and multilingual baselines across IndoNLG tasks, using task-specific automatic and human evaluation. IndoBART and IndoGPT are compact pretrained systems built for the three target languages.
- The benchmark compares scratch training, IndoBART, IndoGPT, mBART, mT5, and task-specific prior models.
- IndoBART is an encoder-decoder model pretrained on Indonesian, Sundanese, and Javanese, while IndoGPT is a decoder-only model pretrained on the same languages.
- IndoBART pretraining uses masked-span recovery, whereas IndoGPT uses an autoregressive language-modeling objective.
- The experiments limit sequence lengths to 512 and select the best learning rate through a specified hyperparameter search.
5 Evaluation Procedure
The evaluation combines automatic task-specific metrics with human judgments across eight tasks and six generation models. Human evaluation uses task-specific criteria for translation, summarization, and chit-chat.
- Automatic evaluation uses SacreBLEU for machine translation, ROUGE for summarization, and F1 plus exact match for question answering.
- Human evaluation covers En ↔ Id, Su ↔ Id, Jv ↔ Id, Liputan6 Xtreme, and XPersona across the benchmark’s generation settings.
- Human evaluation samples 100 test inputs per task and compares six systems, including ground truth, Scratch, mBARTLARGE, mT5SMALL, IndoBART, and IndoGPT.
- Translation is assessed for fluency and adequacy, while summarization uses coherence, consistency, fluency, and relevance.
6 Results and Analysis
Across automatic and human evaluations, mBARTLARGE generally achieves the strongest overall results, while IndoBART remains competitive and substantially more efficient. Results also show task-, language-, and metric-specific differences, including weak automatic evaluation for chit-chat and imperfect agreement with human judgments.
- mT5SMALL leads most local-language translation settings, while all models translate Su → Id and Jv → Id better than the reverse directions.mBARTLARGE also performs well on Sundanese and Javanese translation despite lacking direct pretraining on those languages.
- IndoBART achieves the highest Liputan6 Canonical and Xtreme scores when excluding the task-specific Koto et al. (2020a) system, while mBARTLARGE leads on Indosum.
- mBARTLARGE outperforms all other models by a large margin on both F1 and exact match in question answering, whereas IndoBART leads the chit-chat task despite uniformly low scores.The authors associate low chit-chat scores with the one-to-many nature of open-domain dialogue.
- Automatic chit-chat scores are limited because multiple valid responses can arise from latent factors such as personality, preference, and culture.The authors therefore argue that human evaluation is more suitable for chit-chat.
- Human evaluation ranks mBARTLARGE, IndoBART, mT5SMALL, IndoGPT, and Scratch in that order, sometimes contradicting automated metrics on local-language translation.The paper extends prior concerns about weak BLEU and ROUGE correlation with human judgments to this language family.
- 31.45 overall score makes mBARTLARGE the best model across tasks, while IndoBART ranks second at 3% lower performance.All pretrained models outperform the nonpretrained Scratch baseline.
- IndoBART uses about 20% of mBARTLARGE’s parameters while running almost 4x faster on CPU and 2.5x faster on GPU.IndoGPT is almost twice as slow as IndoBART but achieves performance similar to mT5SMALL.
- Adding more Javanese and Sundanese data improves IndoBART-v2 on ID→JV translation while maintaining performance on other tasks.The results suggest target-language decoding is more sensitive to corpus size than source-language encoding.
7 Conclusion
The paper introduces IndoNLG, Indo4B-Plus, IndoBART, and IndoGPT for evaluating and supporting generation in Indonesian, Javanese, and Sundanese. The models are competitive with mBARTLARGE while using fewer parameters and enabling faster inference, and the resources are released for reproducibility.
- IndoNLG is presented as the first Indonesian NLG benchmark, covering summarization, question answering, open chit-chat, and three machine translation language pairs.
- Indo4B-Plus provides a clean pretraining corpus for Indonesian, Sundanese, and Javanese and is used to train IndoBART and IndoGPT.
- IndoBART and IndoGPT achieve competitive performance against mBARTLARGE despite using approximately 20% as many parameters.
- IndoBART inference is almost 4x faster on CPU and 2.5x faster on GPU than mBARTLARGE.
- The authors release pretrained models, data, code, and a leaderboard to support reproducibility and benchmark transparency.
Ethical Considerations
The paper examines potential misuse, bias, fairness, and representation risks in its Indonesian language models. It reports bias analyses across gender, ethnic group, and religion, while noting that these analyses provide only a partial picture.
- Misuse of Language Models: The authors identify potential misuse of language models for misinformation, plagiarism, spam, phishing, abuse of legal and governmental processes, and social engineering.They anticipate that faithful or steerable generation could lower barriers to such harmful activities and increase their efficacy.
- Fairness, Bias, and Representation: The authors analyze fairness and bias across gender, ethnic group, and religion using co-occurrence tests and generated Indonesian text.They use IndoGPT outputs and manually identify semantically valid phrases in each category.
- Fairness, Bias, and Representation: The authors state that their bias analysis does not reflect all model biases, but provides a partial picture of models trained on Indonesian web data.They frame the analysis as useful for understanding and mitigating bias rather than as a complete assessment.
- Gender: Women are more often associated with caring and physical descriptions, whereas men are more often associated with confidence, responsibility, and strength.These associations are reported from the model’s gender-descriptive phrases.
- Ethnic Group: The model associates ethnic groups with descriptive phrases reflecting how those groups are sometimes presented in the world.Examples include mystical interests for Javanese people and social empathy for Sundanese people.
- Religion: The model associates religions with common terms related to specific religious traditions, including Islam, Christianity, Buddhism, and Hinduism.The analysis lists religion-related terms found in generated outputs.
D Guideline for Conducting Human Evaluation
The human evaluation samples outputs from six generation systems across eight IndoNLG tasks and scores them with task-specific metrics and model rankings. Results are reported through task-level tables and plots, using averages across three native Indonesian annotators.
- Evaluation Setup: Human evaluation covers eight IndoNLG tasks, six generation systems, 100 test inputs per task, and three native Indonesian annotators per sample.The evaluated systems include ground-truth labels, Scratch, mBARTLARGE, mT5SMALL, IndoBART, and IndoGPT.
- Evaluation Metrics: Machine translation is evaluated for fluency and adequacy, summarization for coherence, consistency, fluency, and relevance, and chit-chat for consistency, engagingness, and fluency.Annotators also rank generated texts from 1 to 6, with 1 indicating the most favorable generation.
- Scoring Procedure: Each task metric uses a 5-point Likert scale, and per-sample scores are averaged across the three annotations before task-level statistics are computed.Overall statistics across tasks are computed from aggregated averaged sample scores.
F Quality and Space Time Analysis
The paper defines an overall cross-task score by averaging selected task metrics and separately measures model inference time under fixed decoding conditions. Inference-time experiments use repeated greedy decoding on both CPU and GPU.
- Overall Quality: The overall IndoNLG score averages SacreBLEU for machine translation, ROUGE-L for summarization, F1 for question answering, and SacreBLEU for chit-chat.The score is intended to enable comparison of model quality across heterogeneous tasks.
- Inference Time: Inference time is measured with greedy decoding using fixed encoder and decoder sequence lengths of 256.The experiment averages results over 100 runs on an Intel i9-7900X CPU and a single GTX1080Ti GPU.
G Fairness and Bias Analysis
The fairness and bias analysis uses Indonesian prompt adaptations and IndoGPT generations to examine gender, ethnic, and religious associations. The paper reports descriptive patterns while emphasizing that the analysis is only a partial view of model bias.
- Analysis Procedure: The analysis generates 1,200 IndoGPT outputs from translated and adjusted prompts covering gender, ethnic group, and religion categories.Generation uses temperature 1.0, top-p 0.9, and maximum sequence length 50, followed by manual extraction of semantically valid phrases.
- Gender: Gender analysis finds women more often associated with caring and physical descriptions, while men are more often associated with confidence, responsibility, and strength.The most biased gender-descriptive phrases and their occurrence counts are listed in Table 22.
- Ethnic Group: Ethnic-group analysis reports associations that reflect how groups are sometimes presented in the world, including distinct descriptive patterns for Javanese, Sundanese, Chinese, Indian, Arabic, and Caucasian groups.The associated phrases are listed in Table 23.
- Religion: Religion analysis identifies associations with terms related to Islam, Christianity and Catholicism, Buddhism, and Hinduism.The most favored religion-descriptive phrases are reported in Table 24.