Source-linked AI summary
One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia
Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, Sebastian Ruder
TL;DR
NLP for Indonesia’s 700+ languages is constrained by limited resources and insufficient understanding of linguistic and dialect diversity. The paper surveys the research landscape, examines dialect robustness, and proposes recommendations; it finds Indonesian NLP underrepresented and under-explored and argues for broader, community-aware development.
Problem
NLP research for Indonesia’s 700+ languages faces limited resources, incomplete documentation, and linguistic and dialect diversity that complicates technology development.
Method
The paper reviews Indonesian and local-language NLP, examines dialect and style variation through a Javanese language-identification experiment, and develops recommendations.
Results
Indonesian NLP is underrepresented and under-explored despite Indonesia’s more than 700 local languages, and the paper documents challenges affecting current NLP systems.
Takeaways & Limitations
Improving NLP for Indonesia and other underrepresented languages requires recommendations spanning metadata documentation, research directions, and community engagement.
Takeaways & Limitations
The Javanese dialect experiment uses annotators from specific cities, and different city selections might yield different results.
Abstract
from arXiv · showhide
NLP research is impeded by a lack of resources and awareness of the challenges presented by underrepresented languages and dialects. Focusing on the languages spoken in Indonesia, the second most linguistically diverse and the fourth most populous nation of the world, we provide an overview of the current state of NLP research for Indonesia's 700+ languages. We highlight challenges in Indonesian NLP and how these affect the performance of current NLP systems. Finally, we provide general recommendations to help develop NLP technology not only for languages of Indonesia but also other underrepresented languages.
1 Introduction
Indonesia combines exceptional linguistic diversity with limited documentation and NLP coverage. The paper frames dialect variation and uneven research resources as central challenges for building useful language technology.
- Indonesia has more than 700 languages across 273 million people and 17,508 islands, but many languages lack documentation, formal teaching, or shared standards.
- 440 Indonesian languages are endangered and 12 are extinct, while 71 of 151 surveyed languages have fewer than 100k speakers.
- Javanese and Sundanese have 84M and 34M speakers, respectively, yet local languages remain poorly represented in NLP research.
- Language technology should be available in native languages and may support mutual understanding and language preservation, subject to careful assessment of local usage scenarios.
- Dialect differences in morphology and vocabulary challenge NLP systems, motivating an overview of Indonesian NLP, its language diversity, system impacts, and recommendations.
2 Background and Related Work
Indonesia’s language landscape spans Austronesian and Papuan families, while multilingual NLP resources remain concentrated on Indonesian and a few local languages. Existing datasets and studies provide partial coverage, but local-language research remains limited.
- 2.1 History and Taxonomy: Indonesia’s languages include more than 400 Austronesian languages and Papuan languages, with Austronesian groups distributed across distinct geographic regions.
- 2.1 History and Taxonomy: Indonesian became the national language and is primary in education, mass media, and government, encouraging local-language speakers to use it with their children.
- 2.2 Efforts in Multilingual Research: Multilingual pretrained models largely focus on high-resource languages; mBERT and mT5 include Javanese, Sundanese, and Minangkabau with comparatively little pretraining data.
- 2.2 Efforts in Multilingual Research: Multilingual datasets cover Indonesian across several tasks, but few include Indonesian local languages; WikiAnn covers Acehnese, Javanese, Minangkabau, and Sundanese.
- 2.2 Efforts in Multilingual Research: Research on Indonesian spans tasks including POS tagging, NER, sentiment analysis, hate speech detection, topic classification, and question answering, with growing benchmark standardization.
- 2.2 Efforts in Multilingual Research: Local-language work includes stemming, POS tagging, parallel corpora, machine translation, character segmentation, language identification, text-to-speech, lexicon induction, and abusive-language detection.
3 Challenges for Indonesian NLP
Indonesian NLP is constrained by scarce and unevenly distributed data, extensive linguistic variation, and limited technological access. These challenges affect both resource development and system performance, particularly across dialects and styles.
- 3.1 Limited Resources: Wikipedia contains more than 3 GB of Italian articles but less than 50 MB of Javanese articles, despite comparable speaker populations.Sundanese has less than 25 MB of articles, while languages with comparable numbers of speakers have more than 1.5 GB.
- 3.1 Limited Resources: Many other Indonesian local languages lack Wikipedia instances and alternative high-quality text sources, especially in the spoken-language long tail.Collecting unlabeled corpora is therefore extremely difficult, while labeled datasets are constrained by few speakers, dialect variation, and absent writing standards.
- 3.1 Limited Resources: Indonesian local languages are severely underrepresented in available corpora, limiting the development of strong pretrained language models.Javanese and Sundanese account for only 0.001% and 0.002% of CC–100, while mC4 contains 0.6M and 0.3M tokens respectively out of 6.3T.
- 3.2.1 Regional Dialects and Style Differences: Regional dialects can differ substantially in vocabulary, with Javanese across central and eastern Java showing more than 50% lexical variation.Jambi Malay also varies across villages, and Javanese includes distinct Ngoko and Krama styles associated with different social contexts.
- Case Study in Javanese: Language-identification systems do not perform equally across Javanese dialects and styles, performing more accurately on Ngoko-Central or Krama data represented in their Wikipedia training data.Assuming equal performance across dialects can therefore create problems when systems detect only certain varieties.
- 3.2.2 Code-Mixing: Indonesian NLP must also handle code-mixing, multiple romanized spellings, and uneven technology access across the country.Code-mixing can occur at the morpheme level, local languages may lack established orthographies, and internet access is concentrated mainly on Java.
4 Opportunities
The paper proposes documentation, research, and community-engagement opportunities tailored to Indonesia’s linguistic diversity, limited resources, dialect variation, and uneven written-language coverage.
- Better Documentation: Regional dialect metadata should document variation, communicate system coverage to users, and inform topics in crawled data.The recommendation applies to Indonesian and other languages, especially those with large dialectical differences.
- Better Documentation: Style and register metadata should record politeness and formality for research on style modeling and style transfer.
- Potential Research Directions: Data-efficient methods should reduce dependence on abundant monolingual data while minimizing computation and data-collection costs.Suggested directions include adaptation, few-shot learning, and learning from related languages.
- Potential Research Directions: Parallel Indonesian–local-language data can support translation, exploit bilingualism and vocabulary overlap, and enable synthetic-data generation.The paper recommends beginning collection efforts as soon as possible despite practical challenges.
- Potential Research Directions: Lightweight architectures, distillation, factorization, and pruning could address costly GPU requirements that hinder adoption by local institutions and industries.
- Potential Research Directions: Because many local languages are rarely written, research should include speech understanding, speech recognition, and multimodality alongside text-focused NLP.
- Engagement with Communities: Since solutions are difficult to generalize across local languages, researchers should collaborate with native speakers, linguists, and local communities.The paper also supports open-science movements to facilitate collaboration and reduce barriers to entry.
5 Conclusion
Indonesia has over 700 local languages but remains underrepresented and under-explored in NLP; the paper highlights these challenges and presents recommendations for Indonesia and other underrepresented languages.
- Indonesia has over 700 local languages, yet Indonesian NLP remains underrepresented and under-explored.
- The paper presents recommendations intended to improve NLP for Indonesian and other underrepresented languages.
A Language Statistics
Although Indonesian languages have many speakers, they receive comparatively less published research than European languages.
- Indonesian-language research remains comparatively lower than European-language research despite Indonesia having more speakers than most European languages.
B Wikipedia Availability
Wikipedia availability varies substantially relative to language populations, with some African and Indigenous American languages even more under-resourced.
- Some African and Indigenous American languages are even more under-resourced when Wikipedia size is compared with speaker numbers.
C Wikipedia Vocabulary Overlap
The paper filters noisy Wikipedia vocabulary with KBBI and compares overlap between Indonesian-language sources and the official Indonesian dictionary. The top-100 words align much more reliably with KBBI than the top 1%.
- Wikipedia vocabulary statistics are filtered using the third edition of KBBI, the official Indonesian dictionary.
- 96% of the top-100 Indonesian words overlap with KBBI, making this set more reliable than the top 1%.
- Minangkabau words in a manually curated bilingual dictionary show 55% overlap with Indonesian words, closer to the top-100 result.
D Dialect Differences
Local Indonesian languages exhibit lexical variation across villages and districts, creating differences that can affect language technology. Available studies document variation in Balinese, Sasak, and Banggai vocabulary.
- Balinese and Sasak lexical variation was studied by comparing translations collected from locals across different villages.
- Tables 8 and 9 present lexical variation for Balinese across South Kuta villages and Sasak across North Lombok villages.
- 23.5% lexical variation was reported among 200 basic vocabulary items in the Banggai district of South Sulawesi.
E Local Language Classification
Language identification systems can misclassify Javanese text as Indonesian, English, or Malay. Shared vocabulary contributes to confusion with Indonesian and Malay, while data-size bias is proposed for English errors.
- Javanese texts are misidentified as Indonesian, English, and Malay in the language-identification results.
- Shared words between Javanese and Indonesian, which is similar to Malay, contribute to these language-identification errors.
- The authors attribute English misclassification to data size bias.
F Indonesian NLP Resources
The paper catalogs corpora, datasets, tools, and resources for Indonesian and local Indonesian languages. Although local-language data are much smaller, the collections can support resource construction through Indonesian as a pivot and widespread bilingualism.
- The paper lists monolingual and parallel corpora, Indonesian datasets, Indonesian tools, and local-language datasets.
- Local-language data volumes are much smaller than Indonesian data volumes.
- Indonesian can serve as a pivot language because of large vocabulary overlap with local languages.
- Resource construction can also draw on the fact that most Indonesians speak both Indonesian and a local language.