Source-linked AI summary
NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, Sebastian Ruder
TL;DR
Few public datasets limit NLP research on Indonesian local languages, so the paper builds a parallel corpus and benchmark spanning 10 such languages. Across sentiment analysis and translation experiments, results vary by language: classical models are competitive in sentiment, and translation between local languages performs better than translation involving English.
Problem
Few public datasets for Indonesia’s many languages limit the global research community’s ability to explore them.
Source: §1 · p. 1Method
The paper creates a human-annotated parallel corpus and benchmark for 10 local languages, using translated and quality-checked Indonesian sentiment data.
Source: §1 · p. 1, §3.2 · p. 3Results
Classical methods are competitive in sentiment analysis, while translation among local languages performs well and translation to or from English is extremely poor.
Source: §5.1 · p. 6Takeaways & Limitations
The authors identify an opportunity to create synthetic local-language datasets by translating from a related high-resource language.
Source: §5.1 · p. 6Coverage is limited to 10 languages with 1,000 samples each because of cost and annotator availability.
Source: §6 · p. 8, Limitations · p. 9
Abstract
from arXiv · showhide
Natural language processing (NLP) has a significant impact on society via technologies such as machine translation and search engines. Despite its success, NLP technology is only widely available for high-resource languages such as English and Chinese, while it remains inaccessible to many languages due to the unavailability of data resources and benchmarks. In this work, we focus on developing resources for languages in Indonesia. Despite being the second most linguistically diverse country, most languages in Indonesia are categorized as endangered and some are even extinct. We develop the first-ever parallel resource for 10 low-resource languages in Indonesia. Our resource includes datasets, a multi-task benchmark, and lexicons, as well as a parallel Indonesian-English dataset. We provide extensive analyses and describe the challenges when creating such resources. We hope that our work can spark NLP research on Indonesian and other underrepresented languages.
1 Introduction
NusaX addresses the shortage of public NLP datasets for Indonesian local languages with a parallel corpus covering 10 languages. The paper uses it to benchmark sentiment analysis and machine translation and to analyze model transfer and efficiency.
1 Introduction
- Few public datasets for Indonesia’s many languages limit global NLP research; NusaX introduces a resource covering 10 local languages.Indonesia has more than 700 spoken languages, and many have received little attention from the NLP community.Source: §1 · p. 1
- The benchmark evaluates Indonesian, multilingual, and classical methods on sentiment analysis and machine translation.NusaX is created by translating an existing Indonesian sentiment dataset with bilingual speakers and additional human-assisted quality assurance.Source: §1 · p. 1
- NusaX is a high-quality human-annotated parallel corpus for 10 Indonesian local languages, with corresponding Indonesian-English data for sentiment analysis and machine translation.The 10 languages are Acehnese, Balinese, Banjarese, Buginese, Madurese, Minangkabau, Javanese, Ngaju, Sundanese, and Toba Batak.Source: §1 · p. 1
- The paper analyzes linguistic and empirical characteristics, cross-lingual transfer, and method efficiency in extremely low-resource settings.The contributions include evaluation in few-shot and full-data settings.Source: §1 · pp. 1–2
2 Focus Languages
NusaX covers 10 local Indonesian languages alongside Indonesian, representing languages spoken across several major islands. The set shares a broad Austronesian classification while including language-specific differences in scripts, structure, and use.
2 Focus Languages
- The 10 focus languages are Acehnese, Balinese, Banjarese, Buginese, Madurese, Minangkabau, Javanese, Ngaju, Sundanese, and Toba Batak.Most have populations exceeding 2 million speakers, yet remain underrepresented in NLP research.Source: §2 · p. 2
- The covered languages are Austronesian within the Malayo-Polynesian subgroup and are spoken across Sumatra, Borneo, Java, Madura, and Sulawesi.NusaX uses Latin script, which has become predominant for all covered languages.Source: §2 · p. 2
- Acehnese is shifting toward Indonesian in urban areas, while Toba Batak is also being replaced by Indonesian in urban and migrant areas.Acehnese is noted for diphthongs and many monosyllabic root morphemes; Toba Batak has verb systems reminiscent of Philippine languages.Source: §2 · pp. 2–3
- Banjarese has 73% lexical similarity with Indonesian and many Javanese loanwords, while Minangkabau is not intelligible with Indonesian despite its Malay classification.Banjarese is written in Arabic and Latin scripts; Minangkabau voice patterns differ between standard and colloquial varieties.Source: §2 · pp. 2–3
- The focus languages vary in structure and social use, including Javanese and Sundanese speech or respect levels and distinct verb and word-order patterns.For example, the passages describe Ngaju word order as similar to Indonesian and Toba Batak as verb-initial.Source: §2 · p. 3
3 Data Construction
NusaX expands an Indonesian sentiment dataset through human translation and additional quality assurance. Its construction combines dialect-aware annotator recruitment, filtering, translation guidance, review, and bilingual lexicon creation.
3 Data Construction
- The corpus expands an existing Indonesian dataset through human translation followed by quality assurance from additional annotators.The source is SmSA, a sentence-level, multi-domain sentiment dataset with more than 11,000 instances; 1,000 samples were selected for translation.Source: §3 · p. 3, §3.2 · p. 3
3.1 Annotator Recruitment
- Recruitment accounts for dialect variation by seeking annotators who speak the same or a similar dialect to support mutually intelligible translations.At least two expert annotators native in each local language and Indonesian were used; candidates translated samples and underwent peer review and training.Source: §3.1 · p. 3
3.2 Data Filtering and Sampling
- The team manually removed abusive language and personally identifying information, then used stratified sampling to select 1,000 balanced-label SmSA instances.SmSA data came from online platforms including Twitter, Zomato, and TripAdvisor.Source: §3.2 · p. 3
3.3 Human Translation
- Translation instructions required preserving sentiment polarity, entities, and complete information, while typography consistency was dropped because it was difficult to assess.Directly retaining informal spellings could produce unnatural translations, so typographical consistency was removed as a criterion.Source: §3.3 · p. 4
3.4 Human-Assisted Quality Assurance
- Annotators reviewed one another’s translations, and edited samples showed word edits were most common while major changes were rare.The reported edit analysis covers 100 edited translations each for Balinese, Sundanese, and Javanese; orthographic changes occurred around 10% of the time.Source: §3.4 · p. 4, Table 1 · p. 4
3.5 Bilingual Lexicon Creation
- The bilingual lexicon translates 400 Indonesian words into 10 local languages and English, yielding 800–1,600 word pairs per Indonesian-to-language pair.The lexicon is augmented with PanLex data.Source: §3.5 · p. 4
4 NusaX Benchmark
The NusaX benchmark covers sentiment analysis and machine translation across Indonesian, English, and 10 local languages. It evaluates classical, local-language, and multilingual models, with sentiment data split into train, validation, and test sets for each language.
4.1 Tasks
- The benchmark covers sentiment analysis and machine translation across 12 languages, including Indonesian, English, and 10 local languages.The sentiment subsets maintain the same label distribution across languages.Source: §4.1 · p. 5
4.1.1 Sentiment Analysis
- Each language’s sentiment dataset contains 500 train, 100 validation, and 400 test examples, for 6,000, 1,200, and 4,800 instances respectively across 12 languages.The dataset contains positive, negative, and neutral sentiment classes.Source: §4.1.1 · p. 5
4.1.2 Machine Translation
- The translation task provides 132 possible parallel corpora, each with 1,000 review-domain examples, and is not English-centric.Each corpus has 500 train, 100 validation, and 400 test instances.Source: §4.1.2 · p. 5
4.2.1 Classical Machine Learning
- Classical baselines include naive Bayes, SVM, and logistic regression for classification, plus copy, dictionary substitution, and Moses phrase-based translation baselines.Classical methods are included because they remain widely used locally and offer efficiency in compute-limited settings.Source: §4.2.1 · p. 5
4.2.2 Pre-trained Local Language Models
- The local-language model baselines include IndoBERT and Sundanese-BERT for understanding, and IndoBART and IndoGPT for generation.The study tests how these models adapt to other languages.Source: §4.2.2 · p. 5
4.2.3 Massively Multilingual LMs
- The multilingual baselines are mBERT and XLM-R for sentiment analysis, and mBART and mT5 for machine translation.The benchmark assesses their applicability to low-resource languages.Source: §4.2.3 · p. 5
5 Results
Across sentiment analysis and machine translation, performance varies with model choice, training data, and language similarity to Indonesian. Zero-shot transfer and multilingual training also show that related-language data can support local-language tasks, with important exceptions and data limitations.
5.1 Overall Results
- Fine-tuned IndoBERTLARGE and XLM-RLARGE generally perform best on sentiment analysis, while logistic regression beats them on Acehnese, Buginese, and Toba Batak.The latter languages are described as distinct from the languages covered by IndoBERT and XLM-R.Source: §5.1 · p. 6
- For translation into Indonesian, copying performs well for similar languages, while dictionary-based word substitution yields reasonable improvement.Minangkabau and Banjarese achieve high BLEU without translation; the passage attributes this to their similarity to Indonesian.Source: §5.1 · p. 6, Table 4 · p. 7
- PBSMT and fine-tuned LMs achieve encouraging translation performance despite limited data, but rare words may be copied or mistranslated.The paper reports extremely poor performance for translation from or to English compared with Indonesian, and notes that high SacreBLEU can coexist with these rare-word deficiencies.Source: §5.1 · p. 6, Table 5 · p. 7
- For translation from Indonesian, dictionary substitution helps especially for Javanese, Minangkabau, and Sundanese, while PBSMT and fine-tuned IndoBARTv2 achieve the best scores across multiple local languages.The reported gains and best scores are attributed to grammatical similarity with Indonesian despite limited training data.Source: §5.1 · pp. 6–7, Table 6 · p. 8
5.2 Cross-lingual Capability of LMs
- In zero-shot sentiment transfer, most languages except Buginese and Toba Batak work effectively as sources, averaging about 70–75% F1 versus 80% in monolingual evaluation.Figure 2 trains on the language shown on the x-axis and evaluates on all languages.Source: §5.2 · p. 7, Figure 2 · p. 7
- English also serves effectively as a zero-shot source for all 10 local languages, while the paper links transfer to language similarity and English’s prevalence in pre-training.The authors conclude that closely related languages can support zero-shot or few-shot classification, and that new datasets for such languages need not be large.Source: §5.2 · p. 7
5.3 Multilingual Capability
- Multilingual sentiment training outperforms monolingual training, and leave-one-language-out training matches target-language training, indicating that related-language data is often sufficient.In leave-one-language-out, training uses all data except that of the test language.Source: §5.3 · p. 8, Table 7 · p. 8
6 Data Collection Challenges
Collecting data required finding local-language annotators outside mainstream platforms, then managing communication and annotation across varied access conditions. Online recruitment could miss potential annotators, while in-person collection was impractical across Indonesia’s islands.
Finding annotators
- Popular platforms such as MTurk did not support the local languages, making it difficult to find local-language-speaking annotators.The authors instead searched local forums and Indonesian networks.Source: §6 · p. 8
- Online recruitment through local networks might miss potential annotators, while in-person workshops were impractical because travel between islands is costly.At least two annotators were needed for each language, limiting coverage to the 10 languages presented.Source: §6 · p. 8
Communication with annotators
- Some annotators used mobile apps or annotated on paper, including one who returned a photograph of a printed sheet.Reasons for mobile annotation included ease of use, lacking a laptop, and better keyboard support for diacritics.Source: §6 · p. 8
- Some annotators were difficult to contact because of college or work commitments; unresponsive annotators had to be replaced.Source: §6 · p. 8
7 Related Work
Prior work includes multilingual parallel corpora, emerging benchmarks, and datasets for Indonesian local languages, but labeled resources for the latter remain limited. Existing local-language datasets address varied tasks and often cover only selected languages.
Multilingual Parallel Corpora
- Multilingual parallel corpora support machine-translation research, and FLORES describes a high-quality translation procedure that NusaX also complements with translation quality control.Source: §7 · pp. 8–9
Emerging Language Benchmarks
- Emerging benchmarks cover underrepresented languages and Indonesian NLP, including resources for NER, NLI, language generation, and machine translation.Source: §7 · p. 9
Datasets for Indonesian Local Languages
- Only a limited number of labeled datasets cover Indonesian local languages, with existing resources spanning selected languages and tasks.Examples include weakly supervised NER, abusive-language detection, speech, sentiment analysis, machine translation, and emotion classification.Source: §7 · p. 9
8 Conclusion
NusaX introduces a parallel corpus and benchmark for 10 low-resource Indonesian languages, with analyses of language similarity and cross-lingual transfer. The authors position it as a resource for future work on underrepresented languages and further dataset expansion.
8 Conclusion
- NusaX is presented as the first parallel corpus for 10 low-resource Indonesian languages.Source: §8 · p. 9
- The benchmark covers sentiment analysis and machine translation in zero-shot and full-data settings.Source: §8 · p. 9
- The study analyzes language similarity from linguistic and empirical perspectives by assessing transferability of Indonesian and multilingual pretrained models.Source: §8 · p. 9
- The authors describe NusaX as a testbed for adaptation and few-shot methods, and envision extending coverage to more languages, tasks, and domains.They also suggest the collection guidelines may motivate more efficient high-quality data collection for extremely low-resource languages.Source: §8 · p. 9
Limitations
NusaX expands Indonesian-language NLP resources through a human-translated corpus, but its coverage and representativeness are constrained by dataset size, domains, and annotator-dependent choices.
Limitations
- The corpus covers only 10 languages and 1,000 samples each, with limited domains, possible group or entity bias, and potential translationese.Fluency is self-declared and unverified for most languages, while dialect choices depend on annotators.Source: Limitations · p. 9
A.1 General Information
- The dataset and its accompanying data statement are separately versioned, with the data statement dated February 2022.The dataset is version 1.0 from May 2022.Source: §A.1 · p. 15
A.2 Executive Summary
- NusaX is a human-translated parallel corpus across 10 Indonesian local languages, with human-assisted quality assurance.The dataset documentation lists version 1.0 (May 2022).Source: §A.2 · p. 15, §A.1 · p. 15
A.3 Curation Rationale
- The source Indonesian sentiment data was manually sampled and edited or filtered to preserve label quality and remove harmful content.Curation checks include sentiment correctness, multiple sentiments, discriminatory or abusive content, political charge, Bandung/Sundacentric content, translatability, and misspellings.Source: §A.3 · p. 15
A.4 Documentation for Source Datasets
- NusaX was created by translating SmSA, an Indonesian sentiment-analysis dataset.SmSA is identified as the source dataset in the documentation.Source: §A.4 · p. 15
A.5 Language Variety
- The dataset documentation identifies 10 local languages spoken in Indonesia as NusaX’s coverage.Table 8 lists the covered languages.Source: §A.5 · p. 15, Table 8 · p. 16
A.6 Speaker Demographic
- The source corpus draws on social media and online forums, and the documentation assumes most users are aged 25–34.The age range is an assumption based on the cited majority-user demographic.Source: §A.6 · p. 15
A.7 Annotator Demographic
- Translators were selected for native or fluent status and at least high-school education, but fluency was self-reported and generally unverified.The documentation reports 28 translators in total.Source: Limitations · p. 9, §A.7 · p. 15
B.1 Sentiment Analysis
The appendix documents hyperparameter selection for sentiment analysis and machine translation, but the supplied passages provide no selected values or comparative outcomes.
B.1 Sentiment Analysis
- Statistical sentiment models use spaCy grid search, while pretrained language models tune batch size and learning rate with other hyperparameters fixed.The best statistical model is selected on the development set.Source: §B.1 · pp. 16–17, Table 9 · p. 16, Table 10 · p. 17
B.2 Machine Translation
- Table 11 documents pretrained language-model hyperparameters for machine translation.The supplied passages do not state the selected settings.Source: §B.2 · p. 17, Table 11 · p. 17
C Dataset Statistics
NusaX’s language statistics show substantial lexical differences despite similar vocabulary-size and frequency-histogram patterns, with overlap varying across language pairs.
C Dataset Statistics
- Vocabulary statistics were calculated after removing punctuation and tokenizing the sentences with spaCy.The analysis compares vocabulary characteristics across languages.Source: C Dataset Statistics · p. 17
- Despite similar vocabulary sizes and frequency-histogram patterns, the most common words vary substantially between Indonesian and the local languages.The vocabulary table reports vocabulary sizes and each language’s top ten words.Source: C Dataset Statistics · p. 17, Figure 4 · p. 19, Table 12 · p. 20
- English has the smallest vocabulary overlap with the other languages, while Buginese and Toba Batak also show low overlap.The overlap comparison uses intersection over union; the paper relates the pattern to language-family distinctions.Source: C Dataset Statistics · p. 17, Figure 3 · p. 18