PaperlayerView exact revision on arXiv

Source-linked AI summary

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

Shaoxiong Ji, Zihao Li, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow

arXiv:2409.17892v3cs.CL

TL;DR

Low-resource languages remain underrepresented in imbalanced multilingual training data, motivating broader continual pre-training. The paper builds MaLA and continually trains Llama 2 7B across 546 languages to create EMMA-500. EMMA-500 improves multilingual performance across several task families, though math and machine comprehension remain limited and newer models outperform it on some benchmarks.

  • Problem

    Imbalanced multilingual datasets leave many low-resource languages underrepresented, limiting language coverage.

    Source: §1 · p. 2

  • Method

    The paper builds the MaLA corpus and continually pre-trains Llama 2 7B on a diverse corpus covering 546 languages to create EMMA-500.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    EMMA-500 improves next-token prediction, commonsense reasoning, machine translation, and text classification across a diverse multilingual evaluation.

    Source: §1 · p. 3, §6 · p. 18

  • Takeaways & Limitations

    Well-curated massively multilingual corpora can expand language-model capabilities, particularly for underrepresented languages.

    Source: §1 · p. 2, §6 · p. 18

    EMMA-500 remains limited on math and machine comprehension and does not surpass newer models such as Llama 3, Gemma 2, and Qwen 2 on some benchmarks.

    Source: Limitations · pp. 18–19

Abstract

from arXiv · show

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.

1 Introduction

Low-resource languages remain underserved because multilingual training data are imbalanced, motivating broader and more diverse continual pre-training. The paper introduces MaLA and EMMA-500, which improve performance across multilingual tasks while retaining gains in code generation.

Motivation and Contributions

  • MaLA expands multilingual continual-pretraining data with diverse domains and 546 training languages, producing the EMMA-500 model from Llama 2 7B.The corpus integrates code, books, scientific papers, and instruction data.Source: §1 · pp. 2–3

Results

  • EMMA-500 has the lowest negative log-likelihood among evaluated 4.5B–13B models, indicating strong intrinsic language-modeling performance.Source: §1 · p. 3
  • EMMA-500 improves commonsense reasoning and machine translation over Llama 2-based and multilingual baselines, often surpassing newer advanced models.The reported improvements span multiple evaluation categories rather than a single benchmark.Source: §1 · p. 3
  • EMMA-500 improves text classification and natural language inference, outperforming all compared Llama 2-based and multilingual models.Source: §1 · p. 3
  • EMMA-500 improves machine reading comprehension over Llama 2 7B but still produces quasi-random results similar to other multilingual baselines.Math and machine reading comprehension remain challenging for the evaluated multilingual models.Source: §1 · p. 3
  • With a carefully curated data mix, multilingual continual pre-training does not necessarily regress code generation and surpasses Llama 2 7B on coding.Source: §1 · p. 3

2 The MaLA Corpus

MaLA is a multilingual corpus built through standardized extraction, normalization, cleaning, and deduplication across heterogeneous sources. Its final scale and longer documents support continual adaptation across hundreds of languages.

2 The MaLA Corpus

  • MaLA compiles heterogeneous multilingual sources into a corpus covering 939 languages, with 546 used to train EMMA-500.The workflow addresses extraction, harmonization, preprocessing, cleaning, and deduplication.Source: §2 · p. 3

2.1 Data Integration

  • The integration workflow loads source data, extracts and harmonizes text and metadata into JSONL, and excludes test data while retaining training and validation data.Source: §2.1 · pp. 3–4

2.1.1 Extraction and Harmonization

  • Preprocessing removes dataset-specific quality problems, but because sources are not examined exhaustively, some undetected issues may remain.Source: §2.1.1 · p. 4

2.1.2 Language Code Normalization

  • Language codes are normalized toward ISO 639-3 while preserving or correcting source labels when varieties, merged languages, families, dialect groups, or spelling mismatches prevent direct conversion.Source: §2.1.2 · p. 4

2.1.3 Writing System Recognition

  • Writing-system recognition samples dataset text and assigns a single dominant script, using fallback information when automated identification fails.The corpus does not classify multiple scripts within one dataset.Source: §2.1.3 · pp. 4–5

2.2 Data Cleaning

  • Data cleaning standardizes whitespace and filters undesirable documents using repetition, character, link, stop-word, and related quality checks.Source: §2.2 · p. 5

2.3 Data Deduplication

  • Deduplication combines MinHashLSH near-duplicate filtering with exact hash-based matching across source data.The near-deduplication stage uses 5-grams and a Jaccard similarity threshold of 0.7.Source: §2.3 · p. 5

2.4 Key Statistics

  • The final corpus contains 939 languages, 546 with more than 100k tokens, and has average documents of 90 tokens versus 19 for Glot500-c.The longer sequences provide more context for continual training, according to the paper.Source: §2.4 · p. 6

3 Data Mixing and Model Training

The training mixture combines multilingual text with curated scientific, literary, instructional, and code data to broaden adaptation while limiting forgetting. EMMA-500 is then produced through large-scale causal-language-model continual training.

3 Data Mixing and Model Training

  • The data mix spans languages, domains, lengths, and styles to improve versatility, generalization, and robustness while reducing overfitting to narrow distributions.Source: §3 · p. 6

3.1 Data Mixing

  • Curated scientific papers and books broaden technical and literary coverage, while high-resource data helps mitigate forgetting during continual training.Source: §3.1 · p. 6
  • Instruction data from xp3x and Aya adds multilingual multitask supervision spanning 277 and 101 languages, respectively.Source: §3.1 · pp. 6–7
  • Table 2 defines the continual-training data mix, distinguishing instruction and monolingual text and using different token-counting conventions for code-related data.Source: Table 2 · p. 7
  • Table 3 organizes evaluation coverage through samples per language, language counts, negative log-likelihood, and accuracy.Source: Table 3 · p. 7
  • Code from The Stack is included to support reasoning and mitigate catastrophic forgetting of programming knowledge, with sampling prioritizing quality and data-science code.Source: §3.1 · p. 7
  • The final mixture balances resource groups by downsampling high-resource languages and upsampling low-resource languages, totaling about 136B tokens.Source: §3.1 · p. 8

3.2 Model Training

  • EMMA-500 is trained with a causal language-model objective for 12,000 steps on 200B Llama 2 tokens using 256 Nvidia A100 GPUs.Source: §3.2 · p. 8

4 Evaluation

EMMA-500 is evaluated across intrinsic prediction and multilingual downstream tasks against Llama 2-based, multilingual, and newer strong baselines. It generally improves multilingual and low-resource performance, while remaining limited on the hardest reasoning and reading-comprehension tasks.

4.2 Intrinsic Evaluation

  • EMMA-500 attains lower NLL than all compared models on both Glot500-c and PBC across languages with different resource availability.The authors interpret this as more efficient compression of massive multilingual text.Source: §4.2 · p. 9, Table 4 · p. 9, Table 5 · p. 10

4.3 Machine Translation

  • EMMA-500 outperforms all models on average for X-to-English translation except in high-resource languages, and its English-to-X advantage grows in lower-resource languages.The larger English-to-X margin suggests particular strength in generating non-English text.Source: §4.3 · p. 10, Table 6 · p. 11, Table 7 · p. 11

4.4 Text Classification

  • EMMA-500 outperforms Llama 2-based models on SIB-200, with a 47.5% average improvement in medium- or lower-resource languages, but trails newer models especially in high-resource languages.On Taxi-1500, it surpasses all Llama 2-based models except MaLA-500.Source: §4.4 · p. 11, Table 8 · p. 12

4.5 Commonsense Reasoning

  • EMMA-500 improves multilingual commonsense reasoning over Llama 2-based and multilingual baselines, including up to a 5% XCOPA gain over TowerInstruct, while remaining below some strongest newer models.It outperforms all compared models except LLaMAX Llama 2 7B Alpaca on low-resource XCOPA, where it matches that model.Source: §4.5 · pp. 12–13, Table 9 · p. 13

4.6 Natural Language Inference

  • EMMA-500 achieves the second-best average XNLI accuracy, slightly behind Llama 3.1 8B, and is also second-best on low-resource languages, slightly behind Gemma 2 9B.XNLI classifies multilingual sentence pairs as entailment, contradiction, or neutral.Source: §4.6 · p. 13

4.7 Summarization

  • EMMA-500 performs comparably to Llama 2-based models on XL-Sum and slightly better than Llama 3 and Gemma, but does not significantly outperform them overall.The 44-language test set limits conclusions about low-resource-language capability.Source: §4.7 · pp. 13–14, Table 11 · p. 14

4.8 Math

  • EMMA-500 improves Llama 2 7B on MGSM: direct prompting reaches 0.1702 accuracy, while CoT raises the Llama 2 7B chat score from 0.1 to 0.3 and slightly exceeds Llama 3 and 3.1 8B.Qwen models remain the strongest on this math benchmark.Source: §4.8 · pp. 14–15, Table 12 · p. 15

4.9 Machine Reading Comprehension

  • EMMA-500 improves Llama 2 on BELEBELE and ARC multilingual, but Llama 2 7B-based models remain near-random on these challenging reading-comprehension tasks and trail recent models.BELEBELE covers 122 languages, while ARC multilingual is machine-translated from English science questions.Source: §4.9 · p. 15, Table 13 · p. 16, Table 14 · p. 16

5 Related Work

Prior multilingual LLM research has expanded language coverage through multilingual corpora, vocabulary adaptation, and continual pre-training. These approaches improve low-resource adaptation but introduce challenges such as catastrophic forgetting and direction-sensitive translation performance.

Multilingual LLMs

  • Multilingual LLMs use multilingual and monolingual data for tasks such as translation and summarization, but English-centered training creates disparities for low-resource languages.Vocabulary extension and continual pre-training are described as strategies for narrowing these gaps.Source: §5 · p. 17

Multilingual Corpora

  • Multilingual corpora differ in scale, language coverage, and source diversity, with resources such as CC100, CC100-XL, ROOTS, Occiglot Fineweb 22, ParaCrawl, and CCMatrix supporting multilingual training.ROOTS emphasizes underrepresented languages using web, book, and academic sources.Source: §5 · p. 17

Continual Pre-training

  • Continual pre-training adapts LLMs to new languages or domains without full retraining, but can cause catastrophic forgetting; replay and learning-rate schedules can mitigate this degradation.Parallel-data continual training has also been studied for translation adaptation.Source: §5 · pp. 17–18

6 Conclusion

The paper introduces MaLA and EMMA-500 to improve language-model adaptation across more than 500 languages, especially underrepresented ones. Results across multilingual tasks support the value of a well-curated, massively multilingual corpus while motivating further work on resource disparities.

6 Conclusion

  • The authors compile MaLA and continually train EMMA-500 to improve performance across more than 500 languages, especially low-resource languages.The model is evaluated on next-token prediction, commonsense reasoning, machine translation, and text classification.Source: §6 · p. 18
  • The reported results show notable gains across diverse multilingual tasks and support well-curated, massively multilingual corpora as a route to stronger multilingual capabilities.The conclusion frames reducing high-resource and low-resource disparities as future research.Source: §6 · p. 18

Limitations

The paper identifies evaluation and deployment constraints while describing MaLA’s novelty and data-selection choices. It also acknowledges that data-mix tuning was not ablated and that some preprocessing questions remain open.

Limitations

  • Translated and machine-generated benchmarks can reflect English-centric topics, translation imperfections, and limited cultural sensitivity, constraining multilingual evaluation.The authors call for large-scale natively created multilingual test sets and note that automated tools cannot capture all nuanced qualities.Source: Limitations · p. 18
  • Human evaluation was omitted because multilingual annotation is subjective, inconsistent, expensive, and difficult to scale for low-resource languages.Automatic tools provide a more scalable and consistent alternative despite their imperfections.Source: Limitations · p. 18
  • EMMA-500 does not surpass Llama 3, Gemma 2, and Qwen 2 on certain benchmarks, while remaining limited in mathematical reasoning and machine reading comprehension.The authors attribute these weaknesses partly to numerical difficulty, translation artifacts, and the contextual demands of machine comprehension.Source: Limitations · pp. 18–19
  • The model is not ready for real-world deployment because it lacks human-preference alignment, safety evaluation, and red-teaming for robustness.Further refinement and adversarial testing are required before practical deployment.Source: Limitations · p. 19

A.1 Novelty

  • The paper’s novelty is a 939-language, diverse corpus and continual pre-training that produces EMMA-500 for multilingual tasks, especially low-resource languages.The corpus combines text types such as code, scientific papers, and instructions, and the model is evaluated on 9 tasks and 15 benchmarks.Source: §A.1 · p. 29

A.2 Data

  • The authors did not perform specific ablations to tune the data mix, instead curating it for diversity and balance across languages and text types.They identify further data-mix exploration as beneficial future work.Source: §A.2 · p. 29

A.3 Evaluation

The evaluation uses fixed shot settings and automated multilingual benchmarks, then compares EMMA-500 with diverse baselines and documents reproducibility choices. The paper also explains corpus-scale design decisions and baseline-comparison limitations.

A.3 Evaluation

  • 3-shot machine translation evaluation was selected from Llama 2 results rather than optimized for EMMA-500 through grid search.The authors note that different shot counts may affect performance across tasks and languages.Source: §A.3 · p. 30
  • Automated metrics enable scalable, consistent evaluation across many languages but can miss creativity, contextual appropriateness, and cultural sensitivity.Generation or metric biases may also produce misleading results.Source: §A.3 · p. 30
  • Safety evaluation was outside the study’s primary focus because the release is a continually pretrained multilingual base model rather than an instruction-tuned end-user model.The authors note that existing safety benchmarks cover only a few languages and may not fit massively multilingual evaluation.Source: §A.3 · p. 30

A.4 Comparison with Baselines

  • Compared with MaLA-500, the corpus contains 939 languages, 74 billion tokens, longer documents averaging 90 versus 19 tokens, and more varied text types.The expanded corpus includes code, books, and scientific papers.Source: §A.4 · pp. 30–31
  • EMMA-500 improves multilingual intrinsic tasks, translation, commonsense reasoning, and text classification across more benchmarks than prior comparisons.The authors report improvements over Llama 2-based models and multilingual baselines, with advantages over Tower attributed to broader low-resource generalization.Source: §A.4 · p. 31

A.5 Others

  • The authors release corpus versions, model weights, code, generations, benchmarks, processing scripts, and training software, although some benchmark data requires permission.They also provide per-language results and generation outputs for alternative analyses.Source: §A.5 · pp. 31–32
  • Language-resource groups are defined by MaLA token counts, from high resources above 1 billion tokens to low resources below 1 million.ISO language codes and writing systems support standardized organization and cross-referencing.Source: §A.5 · p. 31
  • The paper does not align its language set with Joshi et al.’s taxonomy because manual alignment would be labor-intensive and peripheral to its core contributions.Instead, it emphasizes broad coverage of underrepresented languages using ISO codes.Source: §A.5 · p. 31

B.1.1 List of Data Sources

MaLA combines many monolingual sources with standardized metadata, language normalization, filtering, deduplication, and resource categorization. The pipeline also applies explicit quality thresholds to programming-language data.

B.1.1 List of Data Sources

  • MaLA aggregates monolingual corpora spanning many languages and domains, while filtering selected high-resource data, translation sources, timestamps, and possible evaluation leakage.The source list includes web, encyclopedic, literary, scientific, code, and language-specific collections.Source: §B.1.1 · p. 32
  • The preprocessing metadata records URL, text, collection, source, and original language code, preserving provenance and text granularity.Sentence- and paragraph-level granularity determines the expected dependence structure among text units.Source: §B.1.1 · pp. 32–33
  • Table 16 enumerates the monolingual datasets used as MaLA source data.The table functions as the source inventory for the corpus construction pipeline.Source: Table 16 · p. 33

B.2 Code

  • Programming-language files are retained under fork-count-dependent thresholds for line lengths and alphanumeric content.The pipeline then applies MinHash deduplication at threshold 0.5 with shingle size 20 and caps each split at 5 million samples.Source: §B.2 · p. 34

C.1 Supported Languages

  • MaLA classifies languages into five token-count tiers, with “unseen” languages excluded from EMMA-500 training.The tiers range from high resources above 1 billion tokens to low resources below 1 million tokens.Source: §C.1 · p. 34

C.2 Data Analysis

The corpus analysis checks writing-system consistency and source composition. It finds mostly language-specific Unicode usage and a broad source mixture dominated by common-crawl-derived collections.

C.2 Data Analysis

  • Most languages have tokens in their own Unicode blocks, although the distribution also reflects code-mixing.The analysis uses Unicode-block percentages to assess language-code conversion and writing-system recognition.Source: §C.2 · p. 34, Figure 3 · p. 36
  • Common Crawl-derived collections are the main source of MaLA documents, while many other sources contribute smaller portions.Large document counts also come from Earthlings through Glot500-c, and web-crawled metadata is grouped by extracted domain.Source: §C.2 · p. 34, Figure 4 · p. 36
  • Figure 2 reports the number of texts and tokens across MaLA’s resource groups.The visualization summarizes corpus scale by language-resource category.Source: Figure 2 · p. 35

D Evaluation Setup

The evaluation setup specifies prompting templates for machine translation and multilingual topic classification tasks.

D.1.1 Machine Translation

  • Machine translation uses a template containing source language, target language, and source sentence fields.Source: §D.1.1 · p. 36

D.1.2 Text Classification

  • The SIB-200 classification prompt presents topic examples before asking for the topic of a news text.Source: §D.1.2 · p. 37
  • Taxi-1500 uses a parallel topic-classification template with religion-related categories and verse-specific input.Source: §D.1.2 · p. 37

E Detailed Results

The detailed-results section covers multilingual benchmarks and introduces PolyWrite for open-ended generation, while noting that such evaluation remains difficult to measure reliably.

E Detailed Results

  • The detailed evaluation covers multilingual reasoning, translation, classification, summarization, open-ended generation, and writing benchmarks, with results and generations released for several tasks.Source: E Detailed Results · p. 37

F Open-Ended Generation

  • Open-ended generation results are moved to the appendix because automated metrics, LLM judges, and reward models have important reliability limitations in multilingual low-resource settings.Source: F Open-Ended Generation · p. 37

F.1 Evaluation Benchmarks

  • PolyWrite evaluates open-ended writing across 240 languages using translated prompts, back-translation quality checks, and BLEU filtering.Source: §F.1 · p. 37
  • PolyWrite contains 51 writing tasks with task-level prompt counts documented in Figure 5 and language-level mean BLEU scores shown in Figure 6.Source: §F.1 · p. 37, Figure 5 · p. 38, Figure 6 · p. 38

F.2 Evaluation Results

EMMA-500 achieves strong open-ended generation results across resource settings, but its high similarity scores indicate reduced diversity and current metrics do not capture all relevant qualities.

F.2 Evaluation Results

  • EMMA-500 maintains high BLEU performance across high- and low-resource settings, while its low-resource Self-BLEU score of 5.09 is highest among evaluated models.Source: §F.2 · p. 38, Table 27 · p. 44
  • EMMA-500 sustains high performance across varying levels of resource availability.Source: §F.2 · p. 39
  • PolyWrite is a 240-language benchmark with 31 writing tasks and 35,751 prompts, created using translated and back-translated prompts filtered by BLEU quality.Source: §F.2 · p. 39
  • Higher Self-BLEU scores for EMMA-500 in medium-low and low-resource languages indicate less diverse generated content than several comparison models.Source: §F.2 · p. 39
  • BLEU and Self-BLEU cannot fully capture cultural relevance, response appropriateness, and other nuanced qualities of multilingual open-ended generation.Source: §F.2 · p. 39

Related papers

Loading 2409.17892v3…