SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

1 Introduction

• SEA datasets are scarce in common pretraining resources, and other available datasets are often scattered, insufficiently documented, or inconsistent in quality and format.Common Crawl covers 2.36% of SEA languages in 11 languages, while C4 covers 10.62% in 11 languages; Common Voice includes six SEA indigenous languages and LAION-5B includes 12.Source: §1 · pp. 1–2
• Prior evaluations collectively cover fewer than 10 SEA languages, leaving the performance of current models on most of the region’s languages largely unknown.The region has more than 1,300 languages.Source: §1 · p. 2
• Anglocentric training data may contribute to cultural bias, and reported failures to reflect local values raise concerns about natural, high-quality generation in SEA languages.The passage also warns that language-support disparities risk marginalizing minority groups.Source: §1 · p. 2
• SEACrowd centralizes and standardizes nearly 500 corpora in nearly 1,000 SEA languages across text, image, and audio, and benchmarks 38 languages on 13 tasks.The study also finds that existing LLM outputs resemble translationese more than natural data in nine SEA languages, and offers strategies for future AI development in the region.Source: §1 · p. 2

2 SEACrowd

• Contributors submit detailed datasheets for public datasets, and submissions are manually checked and corrected for accuracy.The datasheets record dataset descriptions, tasks, languages, licenses, access, annotation, publications, and data splits.Source: §2 · p. 2
• Standardized wrappers support diverse data structures and modalities, while provenance metadata records licensing; only 38.4% of consolidated sources were originally hosted on Hugging Face.The team also engaged data owners to convert three private datasets into public ones.Source: §2 · p. 2
• The initiative consolidates 498 datasheets and provides standardized dataloaders for 399 datasets, covering approximately 1,000 SEA languages.The effort was developed collaboratively, primarily by researchers and engineers based in Southeast Asia.Source: §2 · p. 2

2.1 SEACrowd Catalogue & Data Hub

• The Catalogue stores dataset metadata, while the Data Hub provides standardized dataloaders and the library for schemas and configurations.The platforms share datasheet and dataloader information to support discovery and use.Source: §2.1 · p. 3

2.2 Datasets in SEACrowd

• About 81% of the 498 datasets are textual, ∼8% are vision-language, and ∼11% are speech; the collection spans 83 tasks across NLP, vision-language, and speech.The tasks are standardized into 20 dataloader schemas, and around 53% of datasets have a commercially permissive license.Source: §2.2 · p. 3

3 SEACrowd Benchmarks

• The benchmark suite selects manually annotated or validated subsets from SEACrowd to evaluate existing LLMs, VLMs, and speech models.Evaluations cover models from varied architectures and training approaches.Source: §3 · p. 3

3.1 Datasets

• NLU covers seven tasks in English and 33 SEA languages, NLG covers several generation tasks in 27 SEA languages, ASR covers 15, and image captioning covers four.The NLU benchmark uses 131 subsets; NLG uses 100, ASR uses 19, and image captioning uses four subsets. The VL evaluation is narrow because few VL datasets are available and even fewer have human annotations.Source: §3.1 · pp. 3–4

3.2 Baselines

• The evaluated baselines include commercial and open-source LLMs, multilingual and SEA-focused speech models, and English-trained or multilingual vision-language models.LLM baselines include two commercial models and 17 open-source models; speech baselines include multilingual pretrained and language-finetuned systems.Source: §3.2 · pp. 4–5

3.3 Experimental Settings

• All evaluations are zero-shot, using weighted F1 for NLU, chrF++ and ROUGE-L for NLG, CIDEr for image captioning, and WER or CER for speech recognition.NLU uses three English prompt templates per task; NLG uses one, while image captioning also uses a prompt in the respective SEA language.Source: §3.3 · p. 5

4.1 State-of-the-Art Models on SEA languages

• AYA-101 has the best zero-shot NLU performance, commercial baselines reach a median of ∼0.6 weighted F1-score, and abstractive/extractive QA and summarization scores are poorer.Sailor and SeaLLM are competitive, while English- and country-specific baselines generally perform less effectively.Source: §4.1 · p. 5
• AYA-101 and mT0 show higher equality across the studied SEA languages than Command-R and GPT-4, despite the latter models’ competitive performance.The trend holds across different τ values.Source: §4.1 · pp. 5–6
• Whisper v3 performs better on national languages, while Seamless M4T v2 is more balanced; fine-tuned Whisper models improve on seen languages while retaining other SEA-language capabilities better than Wav2Vec2 XLSR and XLS-R.Whisper performs less optimally on other indigenous languages.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2 Generation Quality in SEA Languages: Translationese vs. Natural Language

• SEA-focused models produce natural sentences over 35% of the time versus under 20% for broad-coverage models; SEA-LION reaches 57.71%, but Tagalog reaches only 11.58%.Burmese and Malay natural-sentence rates are 19.47% and 22.24%, respectively.Source: Table 2 · p. 6, §4.2 · p. 7

4.1 State-of-the-Art Models on SEA languages

• English-only VLMs struggle to generate adequate SEA-language captions, while PaliGemma and mBLIP do better in Thai and Filipino when prompted in those languages.The figure reports CIDEr for image captioning.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Generation Quality in SEA Languages: Translationese vs. Natural Language

• The translationese classifier, fine-tuned from mDeBER-TaV3, achieves 79.08% accuracy across nine languages, using 49 training and 62 test subsets.The subsets contain approximately 39.9k training and 51.5k test sentences.Source: §4.2 · p. 7

5.1 Resource Gaps in SEA

• SEACrowd covers 980 of 1,308 SEA languages, yet over 700 have only one or two datasets and about 90% lack both speech and vision-language datasets.Sentiment data covers 22 languages and named-entity-recognition data covers 17.Source: §5.1 · pp. 7–8
• Only 6% of datasets report detailed quality metrics, while about 70% of curated evaluation subsets lack cultural relevance.Although 78.7% of datasets appear in peer-reviewed venues and 62.4% were fully manually checked, the authors caution that these are surface indicators of quality.Source: §5.1 · p. 8

5.2 Conclusion & Future Work

• The authors define potential utility as the gap between current and ideal utility, and identify national languages plus Javanese, Sundanese, and Hmong as priorities.They highlight naturalness gaps in Malay, Burmese, and Filipino outputs.Source: §5.2 · p. 8
• Resource equity measures the gap between existing and ideal resource availability; the authors identify shortfalls in local languages including Northeastern Thai, Hmong Do, and Cebuano.They state that narrowing these gaps would help preserve languages and continue speakers’ cultural heritage.Source: §5.2 · p. 8
• Figure 6 ranks SEA languages by the descending area of missing potential, showing prioritization based on current utility and resource availability.The authors call for investment in regional language capability, open data collection, community collaboration, and sustainable strategies.Source: §5.2 · pp. 8–9, Figure 6 · p. 9

• Many dialects are absent from the evaluation benchmark; for Malay, it includes Standard Malay but not dialects such as Sarawak Malay.Source: Limitations · p. 9
• Most datasets lack code-switched text, and evaluation coverage varies from 34 languages for NLP tasks to four for vision-language tasks.Source: Limitations · p. 9

• The benchmark uses publicly available resources, obtains explicit consent from contributors where applicable, and follows the resources’ licenses.Source: Ethics Statement · p. 9

A Key Takeaways of SEACrowd

• SEA-specific models AYA-101 and mT0 perform strongly on zero-shot tasks, while Whisper v3 struggles with indigenous languages and current VLMs produce low-quality SEA-language captions.Seamless M4T v2 offers more balanced speech results across SEA languages; abstractive QA and summarization remain difficult for existing models.Source: A Key Takeaways of SEACrowd · p. 20
• SEA-LION v1 produces natural sentences only 57.71% of the time, and Tagalog, Burmese, and Malay outputs often sound unnatural.Source: A Key Takeaways of SEACrowd · p. 20
• SEACrowd covers 74.9% of SEA languages, but resources are long-tailed and about 70% of evaluated subsets lack cultural relevance.Source: A Key Takeaways of SEACrowd · p. 20
• The authors recommend prioritizing national languages with naturalness gaps and under-resourced local languages, alongside investment in R&D, data collection, and collaboration.Source: A Key Takeaways of SEACrowd · p. 20

B Related Work

• Prior work concentrates SEA NLP resources in higher-resource languages such as Indonesian and Vietnamese, while existing regional initiatives include NusaCrowd and community-based projects.The related-work discussion also describes multilingual benchmarks and notes limited SEA language coverage in several SEA-specific pretrained models.Source: B Related Work · pp. 20–21

C.1 Open Contributions

• SEACrowd invites contributors to submit dataset metadata, build standardized dataloaders, identify private datasets, and open private datasets for cataloguing.Source: §C.1 · pp. 21–22

C.2 Measuring Contributions

• Contributors need 20 points for co-authorship, with bonuses for speech or vision datasets and for languages with fewer existing resources.Source: §C.2 · p. 23

D.1 Timeline

• The contribution call ran from November 1, 2023, to March 31, 2024, for datasheets and May 15, 2024, for dataloaders and private-dataset submissions; Figure 9 counts submissions by type.Source: §D.1 · p. 23, §D.2 · p. 23

E.1 Datasheet Reviewing

• Datasheet review checks availability, duplicates, correctness, regional relevance, splits, and licensing, with reviewers able to request corrections or fix errors.The SOP also guides reviewer notes, points, and approval status.Source: §E.1 · p. 24

E.2 Dataloader Reviewing

• Dataloader review checks metadata, subset implementation, test scripts, coding conventions, and schema adherence, with two reviewers required per submission.The SOP also sets configuration rules for dataset types and multilingual datasets.Source: §E.2 · p. 24

F Schemas in SEACrowd

• Each dataloader has a source schema resembling the original dataset and a seacrowd schema that standardizes structure across similar tasks.Source: F Schemas in SEACrowd · p. 24

F.1 NLP

• NLP schemas cover text classification, text-to-text tasks, sequence labeling, question answering, and conversational or structured data.Examples include token-and-label sequences, question/context/answer fields, and chat input/output fields.Source: §F.1 · pp. 24–26

F.2 Speech

• Speech schemas represent speech-text and speech-to-speech data, as well as single-label and multilabel speech classification.Source: §F.2 · p. 27

F.3 VL

• Vision-language schemas cover image-text data, image classification, image question answering, and video-to-text tasks.Their fields include image or video paths, associated text or labels, and task-specific metadata.Source: §F.3 · pp. 27–28

G.1 Datasets

• The evaluation datasets span NLU, generation, vision-language, and speech tasks, including sentiment, NLI, QA, summarization, translation, image captioning, and speech recognition.The listed sources include both SEA-language datasets and multilingual resources.Source: §G.1 · pp. 28–29

G.2 Baselines

• Baseline selection emphasizes variation in model scale, openness, and SEA-language coverage, while including English-only, multilingual, regional, and country-specific NLP models.The study explores selected publicly available models rather than attempting to acquire and fit every SEA-trained LLM; commercial models are also included.Source: §G.2 · pp. 29–30

G.3 Prompts

• All tasks use zero-shot prompting; NLU explores three prompt styles for selected core tasks, while more compute-intensive NLG and vision-language setups use one style.Vision-language evaluation also tests prompts translated into Filipino, Indonesian, Thai, and Vietnamese.Source: §G.3 · pp. 30–31

G.4 Evaluation Results

• Tables 26 and 27 report NLU and NLG results per language, respectively.Source: §G.4 · p. 31

H.1 Training & Evaluation Data

• Text subsets for the translationese classifier were manually selected and validated using publication, domain, and annotation information, with translated text labeled translationese and human-generated text labeled natural.Source: §H.1 · p. 31

H.2 Experiments

• The translationese experiments distinguish natural, human-translated, and machine-translated samples using classical classifiers and fine-tuned mDeBERTa, including tests that merge label classes.Classical methods include logistic regression, Naive Bayes, and SVM with TF-IDF or bag-of-words features; mDeBERTa uses early stopping based on validation accuracy.Source: §H.2 · p. 31

I Supplementary Details for SEA Language Prioritization

• The language-prioritization analysis ranks 20 SEA indigenous languages by speaker demand alongside current utility or resource availability, using AYA-101 performance for utility and about 500 Korean Hugging Face datasets as the availability reference.Figures 10 and 11 present the prioritizations based on utility and data availability, respectively.Source: I Supplementary Details for SEA… · p. 31, Figure 10 · p. 39, Figure 11 · p. 40

J Contributor Demographics

• The supplementary material reports author affiliation and origin-country demographics.Source: J Contributor Demographics · p. 31

K Languages Under Study

• The language lists provide ISO 639-3 codes, names, regions, and population information.Source: K Languages Under Study · p. 31

L Amount of Contributions by Co-Authors

• Co-authors are ordered by their SEACrowd contribution points.Source: L Amount of Contributions by Co-Authors · p. 32