PaperlayerView exact revision on arXiv

Source-linked AI summary

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, Orhan Firat

arXiv:2309.04662v1cs.CLcs.LG

TL;DR

MADLAD-400 addresses limited high-quality general-domain data for the long tail of written languages by auditing and cleaning CommonCrawl-derived multilingual text. The paper constructs and releases a 419-language corpus, trains multilingual translation and language models, and finds translation models competitive with substantially larger systems while few-shot translation remains weaker than supervised models.

  • Problem

    Most publicly available general-domain multilingual corpora cover fewer languages, while long-tail written languages lack high-quality general data sources.

  • Method

    The paper manually audits CommonCrawl-derived multilingual data, applies filters and language corrections, and trains multilingual machine translation and decoder-only language models using public data.

  • Results

    Translation models up to 10.7B parameters are competitive with significantly larger models on some evaluations, while 8B-model few-shot translation improves with more demonstrations but remains significantly weaker than supervised systems.

  • Takeaways & Limitations

    The released audited corpus and baseline models provide the research community with multilingual resources spanning 419 languages and support more transparent dataset creation practices.

  • Takeaways & Limitations

    Audits were mostly conducted by non-speakers, multilingual cleaning tools are unavailable for all 419 languages, and models were evaluated on only 204 languages.

Abstract

from arXiv · show

We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.

1 Introduction

MADLAD-400 scales CommonCrawl-based multilingual data collection to document-level annotations across hundreds of languages, while emphasizing that web-scale multilingual corpora require manual auditing and filtering. The project releases audited data and evaluates models trained on publicly available multilingual resources.

  • Most public general-domain multilingual corpora contain 100–200 languages, leaving broader language coverage as a dataset-creation challenge.
  • A document-level LangID model trained on 498 languages produces annotations for a 5-trillion-token CommonCrawl monolingual dataset.
  • Manual auditing led to discarding 79 languages, renaming or combining others, and adding preprocessing steps to address web-corpus noise.
  • The released models include multilingual machine translation systems up to 10.7B parameters and an 8B decoder-only language model evaluated on highly multilingual translation sets.
  • Figure 1 compares noisy and clean dataset sizes, with the difference especially pronounced for lower-resource languages.

2 MADLAD-400

MADLAD-400 is built by cleaning and auditing CommonCrawl-derived multilingual text, resulting in noisy and clean releases covering 419 languages. The audit identifies systematic quality problems and informs language removal, relabeling, merging, and targeted filtering.

  • The pipeline collects CommonCrawl snapshots, applies preliminary cleaning and document-level LangID, then performs a language-partitioned self-audit to design noise filters.
  • The release contains a 5-trillion-token noisy dataset and a 3-trillion-token clean dataset, each available in document-level and sentence-level forms.
  • Filtering Out Questionable Content: Questionable-content filtering uses document consistency, capitalization, abnormal lengths, technical characters, and curated regex heuristics, removing documents above 20% questionable sentences or with fewer than five sentences.
  • Self-Audit (Quality Review): The audit keeps plausibly in-language or filterable noisy data but removes languages dominated by noise or represented by 20 or fewer documents.
  • Self-Audit (Quality Review): 79 of 498 annotated languages were omitted, leaving 419 languages in MADLAD-400 after audit-driven filtering and dataset changes.
  • Self-Audit (Quality Review): Audited problems include predominantly religious documents, misrendered text, pornographic content, and boilerplate, motivating additional filters and corrections.
  • Self-Audit (Quality Review): Several datasets were renamed or merged after language errors were found, and languages with fewer than 20 documents were removed.

3 Parallel Data

The paper assembles publicly available parallel data covering 156 languages and 4,124 language pairs, then reviews and filters it before expanding multiway coverage.

  • 4.1B sentence pairs cover 156 languages and 4,124 language pairs in the collected parallel dataset.
  • More than 40% of data was filtered from 396 of 4.1k language pairs during preprocessing.
  • Filtering removed exact duplicate pairs, problematic script or encoding cases, excessive source-target overlap, and sentences with extreme length ratios.
  • 20 source-target pairs per language were reviewed for offensive content, pornography, language-pair correctness, and translation plausibility.
  • Language codes were standardized using BCP-47, with ISO codes and non-default script or region tags included when applicable.
  • 11.9B sentences across 20,742 language pairs resulted after adding n-gram-matched multiway data to the processed and publicly available data.

4 Experiments

The experiments evaluate multilingual encoder-decoder and decoder-only models across WMT, Flores-200, NTREX, and GATONES translation benchmarks. The supervised models are competitive with larger systems in several settings, while few-shot translation from the 8B language model improves with demonstrations but remains weaker than supervised models.

  • Multilingual machine translation models: The study trains 3B, 7.2B, and 10.7B multilingual encoder-decoder models sharing parameters across language pairs.A shared 256k-token SentencePiece model is used on both encoder and decoder sides, with a target-language token prepended to each source sentence.
  • Training objectives: Supervised translation and MASS-style monolingual objectives are sampled with equal probability, using UniMax to sample languages from the imbalanced data.The experiments also explore back-translation by sampling monolingual data.
  • Evaluation datasets: The models are evaluated on WMT, Flores-200, NTREX, and GATONES using BLEU and chrF metrics.Flores-200 evaluation covers a computationally constrained 272-pair subset for non-English-centric directions.
  • Supervised translation results: On WMT, the 7.2B and 10.7B models are competitive with the significantly larger NLLB-54B model.WMT and NTREX are news-domain evaluation sets.
  • Supervised translation results: On Flores-200, the model is within 3.8 chrF of NLLB-54B, but the 10.7B model trails by 6.5 chrF on xxyy pairs.Back-translation improves en2xx translation by 3.0 chrF in the Flores-200 case reported.
  • Few-shot language-model evaluation: Few-shot translation with the 8B language model improves as demonstrations increase but remains significantly weaker than models trained on supervised data.The model is evaluated with randomly sampled demonstrations in a source-target prompting format.

5 Training Data Extraction and Memorization

The paper adapts memorization testing to translation, where valid outputs are more constrained than language-model continuations, and finds that translation models can reproduce training data beyond the prompt. Memorization is lower for translate_diff than translate_copy, while many tested languages show none.

  • Testing memorization: Translation memorization tests are difficult because translation has a smaller space of valid outputs than language modeling.The paper modifies standard extraction testing to identify additional leaked data rather than merely reproducing information already available to the adversary.
  • Operational definition: In translate_copy, memorization occurs when a generation longer than the prompt matches the training continuation, with S − P representing additional leaked bits.For translate_diff, the output length is selected from relative continuation and prompt lengths because exact alignment is unavailable.
  • Testing memorization: 2,000 sequences per language are tested with P = 50 using both verbatim matching and approximate matching requiring 90% Levenshtein similarity.The translate_diff setting uses a length-adjusted S to account for differences between source and target continuation lengths.
  • Results: 257/370 languages showed no translate_copy memorization and 130/146 showed none in translate_diff, although some lower-resource languages had higher rates without a strong resource-level correlation.The reported tests examine 50 additional tokens of potential information leakage.
  • Results: Translate_diff showed much lower memorization than translate_copy, which the authors hypothesize reflects the greater difficulty of the task.Approximate matching indicates that paraphrased memorization can produce higher rates than exact matching.
  • Open questions: Determining when memorization is intended or beneficial versus undesired remains an open question.The authors include canaries to support future analysis of memorization in multilingual and machine-translation settings.

6 Related Work

Prior work has produced multilingual corpora and models from general web data, specialized domains, and language-focused initiatives. Coverage ranges from broad general-purpose datasets to resources targeting particular regions, language varieties, or domains.

  • General multilingual corpora: mC4, CC-100, OSCAR, ROOTS, and related resources provide general-purpose multilingual corpora spanning 46 to 166 natural or supported languages.These datasets primarily mine or compile broad web-domain resources for multilingual language modeling and translation.
  • General multilingual corpora: Glot500-C covers 511 languages, while Bapna et al. report an internal dataset covering more than 1,500 languages.The passage notes uncertainty about how many Glot500-C languages consist solely of religious texts.
  • Language-focused resources: Specialized resources include Serengeti for 517 African languages and language varieties, IndicTrans2 for 22 scheduled Indian languages, and a 350+ language children’s-books dataset.These efforts address underrepresentation or domain-specific coverage beyond general multilingual corpora.

7 Limitations

MADLAD-400’s auditing and evaluation remain constrained by limited language expertise, incomplete tooling, sparse benchmarks, and long-tail domain skew. These limitations leave undetected noise and restrict assessment beyond machine translation.

  • Dataset quality: Most audits were conducted by non-speakers, so machine-generated or disfluent content could remain undetected.The limitation directly affects the ability of the self-audit to identify several types of language-specific noise.
  • Dataset quality: No toxicity detectors, classifiers, or filters reliably cover all 419 release languages, limiting dataset cleaning and documentation.The authors encourage users to report remaining issues for listing on the project GitHub page.
  • Evaluation coverage: Models could be evaluated on only 204 MADLAD-400 languages because multilingual evaluation sets are scarce.This constrains evidence about model performance across the full dataset language range.
  • Evaluation coverage: The authors did not conduct non-translation NLP evaluations because available benchmarks cover only 30–50 languages and mostly omit tail languages.They leave these evaluations to future work, particularly for the tail languages that are MADLAD-400’s focus.
  • Data distribution: The long tail is skewed toward specific domains such as religious texts, motivating language-specific corpora and cleaning practices.This domain concentration limits how broadly the web-crawled data may represent long-tail language use.

8 Conclusion

MADLAD-400 contributes a highly multilingual, general web-domain, document-level dataset built through iterative auditing and filtering, alongside released translation and general NLP models. The authors argue that transparent dataset creation can support more responsible data use and inclusive language technology.

  • Dataset contribution: MADLAD-400 is a highly multilingual, general web-domain, document-level text dataset covering 419 released languages after auditing samples from 498 languages.The creation process includes iterative audits, filters, language-code changes, and removal of spurious datasets.
  • Broader contribution: The paper documents dataset creation and auditing in detail to encourage reproducible inspection and improve visibility into downstream data use.The authors connect this visibility with responsible data use and model development within the supported scope.
  • Models and evaluation: Using MADLAD-400, the authors train and release large machine-translation and general-NLP models and evaluate them on highly multilingual datasets.The release is intended to support research toward language technologies covering greater linguistic diversity.

9 Ethics Statement

MADLAD-400 is intended to mitigate the lack of high-quality general data for long-tail written languages, but its use requires careful consideration because preprocessing coverage differs across languages.

  • MADLAD-400 is an audited and cleaned corpus intended to address limited high-quality general data for many written languages.
  • Preprocessing cannot fully remove porn, toxic content, PII, copyrighted content, or noise for languages lacking suitable tools.
  • Practitioners should consider their target use case before using MADLAD-400.

A.1 LangID Details

The LangID appendix describes a 498-language semi-supervised model and documents the dataset’s language, script, and data-volume annotations, including languages removed after auditing.

  • The SSLID model was trained on 498 languages using a temperature of T=3 and additional web-crawled training data.The temperature was used to reduce over-triggering on low-resource languages.
  • The final 79 languages were detected by the LangID model but omitted after the self-audit and therefore have no associated data entries.
  • The tables include languages across a wide range of corpus sizes, from billions of documents or sentences to very small collections.
  • Several entries use script-specific or regional BCP-47 variants, including Arabic-, Latin-, Cyrillic-, and other script labels.
  • The dataset tables record BCP-47 codes, names, scripts, and noisy and clean document, sentence, and character counts.

A.2 Filtering Details

Filtering combines heuristic substring and regular-expression rules with script-specific correction code to remove recurring low-quality or undesirable monolingual text.

  • Cursed Substrings: Cursed substrings and regexes filter recurring artifacts such as boilerplate, pornography, malformed text, and highly regular sequences.The listed patterns include lorem ipsum, download-related text, technical markers, and repeated punctuation.
  • Cursed Substrings: The rules target low-quality sentences ending in pipes, list-like capitalization, and unusually regular text such as shipping labels or country codes.
  • Cursed Substrings: The filtering implementation checks whether any cursed substring occurs in a sentence.
  • Virama Correction: Virama correction substitutes spacing patterns around listed virama characters before or during preprocessing.
  • Chinese Porn Filter: A Chinese pornography filter uses a dedicated list of Chinese and transliterated signal strings.

A.3 Other issues fixed after the self-audit

Native-speaker review and corpus inspection led to targeted filtering, removals, renamings, and merges for languages with suspected noise, boilerplate, or labeling errors.

  • Consulting Language Speakers: Native speakers audited selected languages when the authors suspected noisy or spurious data but could not assess quality confidently.
  • Consulting Language Speakers: The audit removed en_Arab and tly_IR after identifying boilerplate, while zh and zh_Latn received special filters and fa and bho were unchanged.
  • Language Renames and Merges: Corpus inspection found languages whose data differed from LangID predictions, prompting language-code corrections.
  • Language Renames and Merges: dty was renamed zxx-xx-dtynoise, fan was renamed bum, cjk was merged into gil, bjj was merged into awa, and ss-SZ was renamed ss.

A.4 Monolingual Data Details

The self-audit records language-specific quality issues and documents the filtering, removal, and review decisions applied to MADLAD-400.

  • Audit Documentation: Table 10 collects sample-level audit notes, including notes from earlier rounds and exclusions from Round 0.Round 0 notes used to identify cursed substrings were not retained in the table.
  • Audit Findings: Some partitions were renamed or flagged when the audited data represented a closely related language rather than its original label.Bulu data was renamed from Fang, while other entries received warnings about dialects or uncertain language identity.
  • Audit Findings: Audit notes identify unusable, noisy, repetitive, mixed-language, and potentially misclassified web data across individual language partitions.Examples include Spanish in a purportedly non-Spanish partition, Lorem Ipsum, translationese, pornography, and dialect mixing.
  • Filtering Decisions: The audit notes preserve concrete recommendations such as removing short or repetitive documents, filtering boilerplate, and reviewing questionable language assignments.Several recommendations are marked for later filtering or specialist review rather than immediate removal.

A.5 Parallel Data Details

The parallel-data collection combines multiple publicly available sources into a multilingual training corpus spanning many languages and language pairs.

  • Dataset Composition: 4.1B sentence pairs across 157 languages were obtained after preprocessing, with language-pair sizes ranging from 280.3M for en-es to 7,959 for zu-en.The associated language-pair list and data counts are released with the model checkpoints.
  • Data Sources: Table 11 lists the data sources used to create the parallel training data, together with their available languages and language pairs.For NewsCommentary v14, only Kazakh data is used.

A.6 Language Codes

The paper documents language-code conventions and code changes alongside statistics for the preprocessed multiway parallel data.

  • Code Changes: The code-change list includes language-specific mappings with associated sentence-pair counts for the preprocessed data.Examples include Nepali, Kashmiri, Norwegian Nynorsk, Buginese, Azerbaijani, and Guarani.
  • Dataset Statistics: 11,944,961,985 total sentence pairs are reported for the preprocessed parallel data.The total is presented as the overall number of sentence pairs.
  • Code Conventions: Language-code conventions distinguish labels such as Filipino/Tagalog, Twi/Akan, Norwegian Bokmål, Pashto, Malay, and Albanian.The conventions sometimes use macro or conventional codes rather than the more specific or alternative codes used elsewhere.

A.7 Multiway Data Details

The paper trains multilingual translation and language models using MADLAD-400 and public parallel data, evaluates them across highly multilingual benchmarks, and tests memorization with canaries.

  • Data and models: 11.9B sentence pairs cover 19.7k language pairs in the multiway parallel dataset.
  • Data and models: 3B, 7.2B, and 10.7B parameter multilingual translation models are trained with shared parameters across language pairs.The models use a shared 256k-token SentencePiece model across encoder and decoder sides.
  • Data and models: Supervised translation and MASS-style monolingual objectives are sampled with equal probability during multilingual model training.UniMax sampling is used within each task for the imbalanced language distribution.
  • Data and models: An 8B decoder-only language model is trained with a decoder-only adaptation of the UL2 objective.
  • Evaluation: The models are evaluated on WMT, NTREX, GATONES, and Flores-200 using language-pair-level chrF and SacreBLEU scores.NTREX evaluation of LM-8B uses a 50% data split and is not comparable to the machine-translation-model evaluations.
  • Memorization evaluation: 1,279,635 canaries are distributed across languages to test memorization in the monolingual pretraining corpus.Canary types include shuffle, interleave, and training_prefix, with rates adjusted to language resource levels.
Loading 2309.04662v1…