Source-linked AI summary
SeaLLMs -- Large Language Models for Southeast Asia
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, Lidong Bing
TL;DR
Existing LLMs favor high-resource languages, leaving Southeast Asian languages with weaker data support and performance. SeaLLMs address this through SEA-focused continued pre-training, vocabulary expansion, multilingual instruction tuning, and self-preferencing alignment, achieving strong results across SEA-language tasks and large gains over ChatGPT-3.5 in several non-Latin languages.
Problem
Existing multilingual LLMs are dominated by English, while lower-resource Southeast Asian languages face data scarcity, tokenization inefficiency, and weaker multilingual instruction following.
Method
SeaLLMs extend pretrained models with SEA-language data and vocabulary, then apply multilingual supervised fine-tuning and self-preferencing alignment.
Results
SeaLLMs outperform comparable open-source models across SEA-language tasks and surpass ChatGPT-3.5 by large margins in Khmer, Lao, Burmese, and other non-Latin languages.
Takeaways & Limitations
SeaLLMs provide lightweight, culturally attuned language models for Southeast Asian languages while preserving or improving performance on English tasks.
Takeaways & Limitations
Coverage is limited to nine representative Southeast Asian languages, and hallucination or degeneration remains in Burmese and Lao, with moderate hallucination also occurring in other common languages.
Abstract
from arXiv · showhide
Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.
1 Introduction
SeaLLMs address the linguistic and instructional shortcomings of multilingual LLMs by specializing models for Southeast Asian languages. Their evaluations report strong performance across SEA-language tasks, including large gains over ChatGPT-3.5 in several low-resource languages.
- SeaLLMs target Southeast Asian languages that lack the dataset support available to widely spoken languages, addressing a performance gap in existing LLM applications.Existing multilingual models may sacrifice high-resource-language performance and lack multilingual instruction-following ability.
- SeaLLM-13B-v1 outperforms ChatGPT-3.5 by large margins in multilingual instruction following for Khmer, Lao, and Burmese.It also eclipses most available open-source LLMs across world knowledge, language comprehension, and generation tasks in SEA languages.
- SeaLLM-7B-v2 surpasses comparable baselines in math and commonsense reasoning while exceeding ChatGPT-3.5 in reasoning for common SEA languages.SeaLLM-7B-v2.5 subsequently improves significantly over SeaLLM-7B-v2 in SEA languages.
- The models are built through continued pre-training with an SEA-focused vocabulary, multilingual fine-tuning, and self-preferencing alignment.The training process combines multilingual pre-training data, English-dominant and balanced multilingual instruction data, and preference optimization without human annotators or more powerful LLMs.
2 Pre-training
SeaLLMs adapt pre-training to Southeast Asian languages through filtered multilingual data, vocabulary expansion, and balanced language sampling. The expanded vocabulary substantially improves Thai token compression while preserving English tokenization efficiency.
- The pre-training corpus combines web, news, Wikipedia, and scholarly sources filtered for Thai, Vietnamese, Indonesian, Chinese, Khmer, Lao, Malay, Burmese, and Tagalog.The data then undergoes cleansing and content filtration.
- Vocabulary expansion recursively merges whole-word and sub-word pieces for new languages, then prunes rare and low-quality tokens before adding them to the tokenizer.The approach targets inefficient segmentation of non-Latin scripts by existing BPE tokenizers.
- Thai compression improves from 4.29 to 1.57, a 2.7-fold increase in text encoded within the same context constraints.English compression decreases by only 0.3%, preserving its tokenization effectiveness.
- Vocabulary expansion applies to SeaLLM v1 and v2 with Llama-2 and Mistral-7B, but not v2.5 because Gemma-7B already has a 250K-token vocabulary.
- Language-specific data streams and dynamically controlled sampling ratios balance multilingual pre-training, with high-quality data re-fed near the end.This final re-feeding readjusts learning toward high-quality data.
3 Supervised Fine-tuning (SFT)
SeaLLMs use multilingual supervised fine-tuning to support understanding, reasoning, instruction following, and dialogue. Because public SFT data is largely English, the pipeline adds local and synthetic multilingual data, balances it with pre-training data, and uses self-generated preference optimization.
- Multilingual SFT data covers text processing, math and logical reasoning, instruction following, and natural dialogue using local data, translations, self-instruction, and prompting.Native speakers were engaged to further validate or improve synthetically generated data.
- SeaLLM-7B-v2 and v2.5 receive substantially more math and commonsense reasoning SFT data generated with SeaLLM-13B-v1 and strong English models.
- A pre-train-and-SFT hybrid stage prevents English-heavy instruction data from overshadowing smaller Southeast Asian language datasets.It combines pre-training data with much of the English SFT data before the final balanced fine-tuning stage.
- Instructions are consolidated into sequences for efficient supervised fine-tuning, with later synthetic multi-turn conversations improving multi-turn capability.
- Self-preferencing uses SeaLLM SFT models to choose between their own responses, removes inconsistent preferences after order swapping, and trains with direct preference optimization.This avoids relying on GPT-4 or other powerful closed-source models for low-resource languages.
4 Evaluation
SeaLLMs are evaluated across multilingual knowledge, instruction following, reasoning, and translation tasks, with later variants generally competitive with or superior to established baselines. Gains are especially pronounced for low-resource, non-Latin SEA languages.
- Sea-bench Peer Comparison: SeaLLM-7B-v2.5 performs on par with or surpasses ChatGPT-3.5 across linguistic and writing tasks, driven largely by gains in Burmese, Lao, Khmer, and Thai.The comparison also includes Qwen1.5-7B-chat.
- Model Variants: SeaLLM-7B-v2 demonstrates strong English MT-Bench performance for its size, while SeaLLM-7B-v2.5 improves substantially across SEA-language capabilities over earlier variants.The evaluation compares closed, open, multilingual, and monolingual models.
- Sea-bench Peer Comparison: SeaLLM-7B-v2.5 exhibits the best multilingual world-knowledge performance given its size and remains competitive with GPT-3.5.Table 2 covers multiple languages and model sizes.
- Math Reasoning: SeaLLM-7B-v2.5 records 78.5 on GSM8K and 34.9 on MATH in English, while exceeding GPT-3.5 on the corresponding SEA-language evaluations.The evaluation uses zero-shot chain-of-thought prompting and translated Chinese, Vietnamese, Indonesian, and Thai tasks.
- Translation: SeaLLM-13B exceeds ChatGPT-3.5 in translation for low-resource languages such as Lao and Khmer while remaining comparable in higher-resource Vietnamese and Indonesian.The evaluation uses 4-shot chrF++ scores on Flores-200.
- Translation: For direct SEA-language translation, SeaLLM-13B-v1 achieves higher chrF++ scores than ChatGPT-3.5 in most cases, especially for low-resource language pairs.Figure 4 encodes score differences as SeaLLM-13B-v1 minus ChatGPT-3.5.
5 Conclusion
The paper presents SeaLLMs as specialized models for equitable, culturally aware language technology in Southeast Asia. It reports strong language understanding and generation, particularly in SEA languages, alongside attention to local norms and legal considerations.
- Conclusion: SeaLLMs combine specialized SEA-language modeling with cultural and legal awareness, challenging dominant models particularly in Southeast Asian languages.The conclusion frames this as a foundation for more inclusive language-model research.
6 Limitations
SeaLLMs cover only a subset of Southeast Asia’s linguistic diversity and still exhibit hallucination and degeneration in some languages. These limitations constrain their reliability and regional coverage.
- Scope: SeaLLMs cover nine representative Southeast Asian languages, leaving hundreds of other regional languages outside the reported scope.Javanese and Tamil are cited as examples.
- Reliability: Despite outperforming popular models in non-Latin low-resource languages, SeaLLMs still show considerable hallucination and degeneration in Burmese and Lao.Moderate hallucination remains possible in other common languages.
A Vocabulary Expansion
SeaLLM expands Llama’s vocabulary for new and low-resource languages by selectively merging compatible tokens from the NLLB vocabulary. The resulting tokens are pruned before inclusion to improve encoding efficiency and avoid disruptive cross-language tokens.
- Token Merging: The vocabulary-expansion algorithm tokenizes seed-language data, recursively merges supported NLLB tokens, and adds intermediate subwords to the Llama tokenizer.This selective merger enriches coverage for new and low-resource languages.
- Token Pruning: Frequency-based pruning removes rare or low-quality tokens before final tokenizer integration, reducing inefficient or cross-language vocabulary additions.The stated goal is to preserve efficient language encoding without disrupting pretrained knowledge.
B Sea-bench Evaluation Details
Sea-bench evaluates models across nine languages and five categories using GPT-4 ratings. SeaLLM-13b surpasses ChatGPT-3.5 in most non-Latin languages, while trailing in Latin-based languages mainly on math reasoning.
- SeaLLM-13b far exceeds ChatGPT-3.5 in Burmese, Lao, and Khmer, but trails it in Latin-based languages, mostly for math reasoning.The comparison uses GPT-4-rated Sea-bench score evaluations.
- Vocabulary extension exhaustively merges consecutive token pairs using a target vocabulary, adds new tokens, updates frequencies, and prunes rare words.The algorithm operates on unlabeled data with a minimum-frequency threshold m.
- GPT-4 rates Sea-bench performance across 9 languages and 5 categories for SeaLLM-13b and baseline models.