PaperlayerView exact revision on arXiv

Source-linked AI summary

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, Orhan Firat

arXiv:2309.04662v1cs.CLcs.LG

TL;DR

Publicly available general-domain multilingual corpora typically cover only 100–200 languages, while high-quality general data sources are lacking for the long tail of written languages. MADLAD-400 builds an audited CommonCrawl corpus spanning 419 languages; its translation models are competitive with NLLB-54B on WMT and the 10.7B model is within 3.8 chrF of NLLB-54B on Flores-200.

  • Problem

    Publicly available general-domain multilingual corpora typically cover only 100–200 languages, while high-quality general data sources are lacking for the long tail of written languages.

    Source: §1 · p. 1, §9 · p. 11

  • Method

    MADLAD-400 uses document-level CommonCrawl text, self-audits samples across 498 languages, applies filters, and releases a 419-language dataset.

    Source: §2 · p. 2, §8 · p. 11

  • Results

    On WMT, the 7.2B and 10B models were competitive with NLLB-54B; on Flores-200, the 10.7B model was within 3.8 chrF of NLLB-54B.

    Source: §4.4 · p. 8

  • Takeaways & Limitations

    The authors conclude that detailed, public reporting of audits and dataset improvements can encourage due diligence in large-scale pretraining-data creation.

    Source: §8 · p. 11

    Audits were mostly conducted by non-speakers, and models were evaluated on only 204 of MADLAD-400’s 419 languages.

    Source: §7 · p. 10, §8 · p. 11

Abstract

from arXiv · show

We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.

1 Introduction

MADLAD-400 scales CommonCrawl-based multilingual data mining with document-level language identification, then manually audits the resulting corpus to address noise and labeling problems. The audit reshapes the dataset, while multilingual translation and language models are trained and evaluated to validate it.

1 Introduction

  • Most publicly available general-domain multilingual corpora contain 100–200 languages, while broader coverage is found in some domain-specific datasets.Examples include religious content, children’s books, and dialects.Source: §1 · p. 1
  • A document-level LangID model covering 498 languages produced a 5-trillion-token monolingual dataset from CommonCrawl.This scales the common approach of mining language-specific data from general web crawls.Source: §1 · p. 1
  • Manual auditing led the authors to discard 79 languages, rename or combine others, and apply additional preprocessing before evaluating multilingual models.They trained machine translation models up to 10.7B parameters and an 8B decoder-only model, evaluating them on highly multilingual translation sets.Source: §1 · p. 1
  • For the lower-resource half of the dataset, the median ratio between noisy and clean dataset sizes is about 4×.The figure notes that the log scale makes the size difference appear smaller than it is.Source: Figure 1 · p. 2

2 MADLAD-400

MADLAD-400 was built from CommonCrawl through language identification, iterative filtering, and manual self-auditing. Its clean release contains 3 trillion tokens across 419 languages; a 5-trillion-token noisy version is also available.

2.1 Preliminary Filters

  • Preliminary filters deduplicated lines across documents [44], removed pages with fewer than three 200-character lines [68], and excluded common boilerplate such as “Javascript” and “lorem ipsum” [57].Source: §2.1 · p. 3

2.2 Language Identification (LangID)

  • A Semi-Supervised LangID model trained on 500 languages assigned document-level labels by majority sentence-level prediction, producing MADLAD-400-noisy.Source: §2.2 · p. 3

2.3 Filtering Out Questionable Content

  • Documents were removed if over 20% of sentences were questionable or they contained fewer than five sentences.Questionable sentences were identified through LangID inconsistency, list-like capitalization, abnormal length, technical characters, or “cursed regexes”; the length heuristic was noted to work poorly for ideographic languages.Source: §2.3 · p. 3

2.4 Self-Audit (Quality Review)

  • The self-audit reviewed 20 documents per language and led to filter changes and language corrections or removals; issues included religious content, misrendered text, pornography, and boilerplate.Languages were retained if plausibly in-language; noisy but filterable data prompted filter notes, while majority-noise or very small datasets were omitted.Source: §2.4 · pp. 3–4, Table 3 · p. 4

2 MADLAD-400

  • 79 of 498 languages were omitted, leaving 419 in the final dataset, released as a 3-trillion-token clean version and a 5-trillion-token noisy version.Both versions are available in document-level and sentence-level forms.Source: §2 · p. 2, §2.4 · p. 3

2.5 Additional Filters

  • Additional filters corrected detached viramas, converted likely Zawgyi documents to Unicode, and blocklisted pornographic Chinese pages.For Zawgyi, conversion applied above 50% detection probability; the Chinese blocklist reduced document count by 17% and file size by 56%.Source: §2.5 · p. 4

2.6 Correcting Other Systematic Issues.

  • Five datasets in the wrong language were renamed or merged, six suspicious languages received native-speaker review, and datasets with fewer than 20 documents were removed.Source: §2.6 · pp. 4–5

3 Parallel Data

The section describes a publicly sourced parallel corpus spanning 156 languages and 4.1 billion sentence pairs, along with its preprocessing, quality review, language-code conventions, and expansion into multiway data.

3 Parallel Data

  • The collected parallel dataset covers 156 languages, 4.1B sentence pairs, and 4,124 language pairs.The data sources and associated language pairs are listed in Appendix A.5.Source: §3 · p. 5

3.1 Filters

  • More than 40% of data was filtered out for 396 of roughly 4,100 language pairs.Preprocessing included exact source-and-target pair deduplication, language-specific virama and encoding filters, overlap and length-ratio checks, and script filters.Source: §3.1 · p. 5
  • Unmatched-toxicity filters from NLLBTeam et al. were unusable for most languages because wordlists flagged non-toxic homonyms, although pornographic content was removed for zh and ko.More than 3% of documents were marked as having unmatched toxicity in each of the listed languages.Source: §3.1 · p. 5

3.2 Self-Audit (Quality Review)

  • Auditors reviewed 20 source-target pairs per language for offensive content, pornography, language-pair correctness, and translation plausibility, which was often judged by guesswork without native speakers.The review covered 157 languages, and full instructions, findings, and dataset changes are provided in Appendix A.5.Source: §3.2 · p. 5

3.3 A Note on Language Codes

  • The dataset uses BCP-47 language codes, with script and region tags included when they are not CLDR defaults; this follows varied source-dataset coding conventions noted by Kreutzer et al. [40].For example, ks denotes Kashmiri in the default Nastaliq/Arabic script, while ks_Deva denotes Kashmiri in Devanagari.Source: §3.3 · p. 6

3.4 Multiway Data

  • Applying n-gram matching with n = 8 from Freitag and Firat [25] to the processed data and public data yielded 11.9B sentences across 20,742 language pairs.Full details are provided in Appendix A.7.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate MADLAD-400-trained machine-translation and language models across translation benchmarks. The machine-translation models are competitive with a much larger model on WMT, while few-shot translation with the 8B language model remains weaker than supervised models.

4.1 MT Models

  • MT experiments train 3B, 7.2B, and 10.7B encoder-decoder models with shared parameters across language pairs and a shared 256k-token SentencePiece model.Training mixes supervised parallel-data machine translation and a MASS-style objective on MADLAD-400 with equal probability; the 7.2B model is also fine-tuned on a 1:1 mix of original and filtered back-translated data.Source: §4.1 · p. 6

4.2 Zero-shot Translation with Language Models

  • The zero-shot translation experiments train 8B decoder-only language models solely on monolingual data using a decoder-only variant of the UL2 objective.The setup follows the training schedule and model configurations of Garcia et al. [27] and prior work [17] [27] [52].Source: §4.2 · p. 6

4.4 Results

  • Within 3.8 chrF of NLLB-54B on Flores-200, the reported model is competitive with NLLB-54B on WMT; on xxyy pairs, the 10.7B model trails by 6.5 chrF.The authors attribute the xxyy gap in part to model size, domain differences, and back-translated data.Source: Table 4 · p. 7, Table 5 · p. 7, §4.4 · p. 8

4.3 Evaluation

  • Evaluation uses sacreBLEU BLEU and chrF on WMT, Flores-200, NTREX, and GATONES, with language coverage restricted by available data and computational constraints.For non-English-centric Flores-200 pairs, the evaluation uses a 272-language-pair subset; GATONES uses the subset overlapping with MADLAD-400 or the parallel training data.Source: §4.3 · p. 7

4.3.1 Few-shot evaluation for language modeling

  • Few-shot evaluation prompts the language model with randomly sampled demonstrations from each dataset’s development split, using 1,000 NTREX examples as its development set.The prompt identifies source and target languages by their English names; the remaining NTREX examples are used for testing.Source: §4.3.1 · pp. 7–8

4.4 Results

  • Few-shot translation scores for the 8B language model rise with more demonstrations, but remain significantly weaker than models trained on supervised data.The paper reports per-language-pair results in Appendix A.10.Source: §4.4 · p. 8
  • 3.0 chrF: back-translation improves en2xx translation in most cases on Flores-200; the 10.7B model is within 5.7 chrF of Bapna et al.’s reported scores.The authors caution that their setup is likely sub-optimal and that comparisons with Bapna et al. [9] are difficult, citing iterative back-translation and more in-house text data as advantages for that work.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 9

5 Training Data Extraction and Memorization

The section adapts training-data extraction tests for translation and finds that translation models can reproduce training data beyond the prompt, while distinguishing extraction from ordinary translation is difficult. It also introduces canaries to support future study of multilingual translation memorization.

The Difficulty of Assessing Memorization in Translation Settings

  • Translation has fewer valid outputs than language modeling, making standard memorization tests difficult to apply; in translate_copy, a matching continuation may already be known to the adversary.The paper notes that membership inference may still be possible.Source: §5 · p. 9

Memorization in Translation Settings

  • For translate_copy, memorization means matching a continuation beyond P; for translate_diff, S scales with relative token lengths, with δ representing additional leaked information.The tests sampled 2,000 sequences per language at P=50 and used verbatim and 90% Levenshtein-similarity matching [32].Source: §5 · p. 9

Discussion

  • Whether translation memorization is intended or beneficial versus undesired remains an open question.Source: §5 · p. 9

Results.

  • At S=P+50, translate_copy had higher extraction rates than translate_diff; 257/370 and 130/146 languages, respectively, showed no verbatim memorization (Figure 2).Some lower-resource languages had higher rates, but resource level showed no strong correlation; approximate matching also indicated higher rates, consistent with paraphrased memorization.Source: §5 · p. 9, Figure 2 · p. 10

Discussion

  • The authors introduce canaries—modified real-data examples designed to resemble natural data yet act as outliers—to measure extraction in multilingual and machine translation settings [6, 33].They generated 1,945,631 canaries across MADLAD-400 and parallel data, approximately 0.0026% of training data.Source: §5 · p. 10

6 Related Work

Related work describes broad multilingual corpora for machine translation and language modeling, alongside efforts focused on languages underrepresented in general corpora.

6 Related Work

  • mC4 [68], CC-100 and CC100-XL [19] [47], OSCAR [2], and ROOTS provide general multilingual corpora spanning from 46 natural languages to 166 languages.mC4 is a general web corpus used to train mT5 for downstream NLP tasks.Source: §6 · p. 10
  • Glot500-C [31] covers 511 languages, though the number consisting solely of religious texts is unclear; other datasets draw on 1500+ languages, CommonCrawl and ParaCrawl, or children’s books [9] [45].Bapna et al. created an internal dataset, NLLBTeam et al. mined data, and Leong et al. created a dataset from children’s books.Source: §6 · p. 10
  • Serengeti [3] provides a dataset and model for 517 African languages and varieties, while IndicTrans2 [26] introduces a machine-translated model for India’s 22 scheduled languages.Source: §6 · p. 10

7 Limitations

The authors’ audits and models remain limited by gaps in language-specific auditing and evaluation, while long-tail data skews toward particular domains. They call for language-specific corpora and cleaning practices to address these gaps.

7 Limitations

  • Non-speaker audits may miss machine-generated or disfluent content, and no toxicity detectors, classifiers, or filters work reliably across all 419 languages.These limitations constrain dataset cleaning and documentation; issues may remain, and users are encouraged to report them on the project GitHub page.Source: §7 · p. 10
  • Only 204 MADLAD-400 languages were evaluated; despite common non-translation NLP evaluation of decoder-only models [7] [5], such tests were deferred because benchmarks cover mostly 30-50 non-tail languages.The authors leave these evaluations to future work.Source: §7 · pp. 10–11
  • Self-audits found long-tail data skewed toward specific domains such as religious texts, motivating language-specific corpora beyond web crawls and language-specific cleaning tools and practices.Source: §7 · p. 11

8 Conclusion

MADLAD-400 is a manually audited multilingual dataset released in 419 languages, with its creation process documented in detail. The authors also train and release machine-translation and general NLP models, aiming to encourage more inclusive language technologies.

8 Conclusion

  • The self-audit sampled all 498 languages, developed filters, and removed spurious datasets, yielding a 419-language release.The paper describes the audit iterations and improvements made to the preliminary dataset.Source: §8 · p. 11
  • The authors argue that detailed, reproducible documentation of dataset creation can encourage manual inspection and support responsible data use.They suggest that greater visibility into dataset creation can improve model development.Source: §8 · p. 11
  • Using MADLAD-400, the authors train and release large machine-translation and general NLP models, which they evaluate thoroughly.They hope this work motivates language technologies that better include humanity’s language diversity.Source: §8 · p. 11

9 Ethics Statement

MADLAD-400 aims to mitigate the shortage of high-quality general data for long-tail written languages, but preprocessing cannot remove all problematic content across languages, so practitioners should assess their use case carefully.

9 Ethics Statement

  • MADLAD-400’s audited, cleaned corpus is intended to help address the lack of high-quality general data for long-tail written languages [37].The authors hope its availability will mitigate this shortage.Source: §9 · p. 11
  • Because tools for removing pornography, toxic content, PII, copyrighted content, and noise are unavailable for some languages, practitioners should carefully consider their target use case.The authors note that this limits how extensively MADLAD-400 can be preprocessed.Source: §9 · p. 11

A Appendix · A.1 LangID Details

The appendix describes a 498-language semi-supervised LangID model trained with temperature-adjusted data and web-crawled examples to improve robustness to web text. It also documents the language inventory and notes that 79 languages the model was trained to detect were omitted after self-auditing.

A.1 LangID Details

  • The Transformer-Base SSLID was trained on 498 languages, with the selected languages roughly corresponding to the 498 with the most sentences in the referenced dataset.The authors direct readers to Table 9 for the complete language list.Source: §A.1 · p. 16
  • Training data was sampled at temperature T=3 to reduce over-triggering on low-resource languages.The training data follows Language ID in the Wild, with modifications to sampling and data sources.Source: §A.1 · p. 16
  • The training set was supplemented with web-crawled data from Language ID in the Wild to improve web-domain robustness and potentially distill its filtering.The added web-crawled data had already passed the filters described in that work.Source: §A.1 · p. 16
  • Table 9 lists MADLAD-400 languages by BCP-47 code, name, script, and associated data amounts.Its inventory spans the language entries shown across the appendix passages.Source: Table 9 · p. 16, §A.1 · pp. 16, 17, 18, 19, 22
  • The LangID model was trained to detect 79 languages that were omitted from MADLAD-400 after self-auditing.These entries have dashes for their data amounts in the language inventory.Source: Table 9 · p. 16, §A.1 · p. 21

A.2 Filtering Details

The filtering details describe cursed substring and regex checks for noisy text, a Chinese porn-signal list, and virama normalization by removing surrounding spaces. They also note common low-quality pipe endings and patterns targeting regularly formatted text.

Cursed Substrings; Chinese Porn Filter

  • Filtering used cursed substrings and regexes for noisy or regularly formatted text, alongside a separate list of Chinese porn signals.The notes flag low-quality sentences ending in a pipe, excluding Devanagari danda text, and regexes targeting A N T S P E A K, List Case, and unusually regular text.Source: §A.2 · pp. 22–24

Cursed Substrings

  • The demonstration filter checks text against the cursed patterns; the authors recommend string inclusion for all but the last four patterns and compiled regexes for those.The listed patterns include replacement characters, spam or adult-content terms, and repeated punctuation or highly regular token sequences.Source: §A.2 · pp. 22–23

Virama Correction

  • Virama correction removes spaces around characters in the listed virama set using a regex substitution.The set covers virama and related characters across multiple scripts.Source: §A.2 · p. 23

A.3 Other issues fixed after the self-audit

The self-audit led to language-specific data filtering and removal, as well as renames and merges where corpus languages differed from LangID predictions.

Consulting Language Speakers

  • Native-speaker audits triggered special filters for zh and zh_Latn, removal of boilerplate en_Arab and tly_IR data, and no changes to fa or bho.The speakers were consulted when the data was suspected to be noisy or spurious but its quality could not be ascertained.Source: §A.3 · p. 24

Language Renames and Merges

  • LangID/URL mismatches led to dty renamed to zxx-xx-dtynoise, fan to bum, cjk merged into gil, bjj into awa, and ss-SZ renamed to ss.The dty corpus consists mainly of mis-rendered PDFs and may be useful for denoising or decoding garbled PDFs.Source: §A.3 · p. 24

A.4 Monolingual Data Details

The monolingual-data appendix records language-by-language self-audit findings, including quality quirks and filtering or release recommendations. Some notes refer to older data versions, while others describe issues in the current dataset.

A.4 Monolingual Data Details

  • Round 2–3 notes mix judgments on current data with comments about older, less-filtered versions, while recording language-specific quality quirks.The appendix points readers to overall dataset statistics in Table 9 and audit notes in Table 10.Source: §A.4 · p. 24
  • Auditors rated languages across a wide quality range, recommending removal or caution for some while identifying good data, biblical material, and language misidentifications in others.For example, the notes identify Altai data and flag Awadhi data as biblical content under a different code.Source: §A.4 · p. 25
  • Notes flag script and boilerplate noise, recommending short-text or script filters and disclaimers where data remains usable despite substantial noise.Some entries also mention pornographic material, English boilerplate, and repeated or malformed text.Source: §A.4 · p. 27
  • Language-specific recommendations include removing poor or noisy data, filtering short or repetitive documents, and correcting virama-related issues.Other entries are marked acceptable, sometimes with cautions about English noise or mixed sources.Source: §A.4 · p. 28
  • The notes mark many entries as acceptable but also identify removals, including boilerplate-heavy or English-dominated data, and flag possible language or script confusion.Several entries are annotated as biblical or religious data, and some are recommended for merging or renaming.Source: §A.4 · p. 29
  • Further audit notes flag mixed dialects, translationese, misrendered PDFs, and noisy or potentially machine-translated text, alongside many entries judged good or acceptable.Some entries are marked for removal or review after filtering; others carry cautions about language identification or community naming.Source: §A.4 · pp. 30–31

A.5 Parallel Data Details

After preprocessing, the parallel training dataset contains 4.1B sentence pairs across 157 languages, with pair counts ranging widely by language pair.

A.5 Parallel Data Details

  • 4.1B sentence pairs cover 157 languages, ranging from 280.3M en-es pairs to 7,959 zu-en pairs.The complete language-pair counts are available with the model checkpoints.Source: §A.5 · p. 31

A.6 Language Codes

This appendix documents language-code choices made in Section 4.1, including alternatives to other resources’ codes and conventions for macro-language and dialect labeling.

A.6 Language Codes

  • 11,944,961,985 sentence pairs make up the preprocessed multiway data, whose target-language counts are listed in decreasing order.Source: §A.6 · p. 32, Table 12 · p. 32
  • The dataset uses fil for Filipino/Tagalog and ak for Twi/Akan, including Fante, rather than tl and tw, respectively.Source: §A.6 · p. 33
  • It uses chm for Meadow Mari instead of the correct mhr, and mrj for Hill Mari.Source: §A.6 · p. 33
  • By convention, it uses no for Norwegian Bokmål, ps for Pashto, ms for Standard Malay, and sq for Albanian without distinguishing Gheg and Tosk.Source: §A.6 · p. 33
  • It uses ber for Tamazight after consultation with speakers, and macro codes qu for Quechua and ff for Fulfulde; qu data usually mixes Ayacucho and Cusco dialects.Source: §A.6 · p. 33

A.7 Multiway Data Details … Maintenance

This section documents MADLAD-400’s curation, scale, purpose, uses, release, and data-quality risks, alongside multilingual model construction, training, and evaluation details. It also describes the canary dataset for studying memorization, including its sampling, processing, distribution terms, and maintenance.

A.7 Multiway Data Details

  • The multiway dataset contains 11.9B sentence pairs across 19.7k language pairs, with pair counts by target language listed in Table 12.Source: §A.7 · p. 33

A.8 Model Training Details

  • MT training covers 3B, 7.2B, and 10.7B models with shared parameters, a shared 256k-token SPM, and equally sampled supervised translation and monolingual MASS-style objectives.The 10.7B model uses dropout p = 0.2 to mitigate overfitting to low-resource languages.Source: §A.8 · p. 33

A.9 Languages Evaluated

  • The appendix lists evaluation languages and per-pair chrF/SacreBLEU scores for WMT, NTREX, GATONES, and Flores-200 variants, plus approximate-memorization extraction analyses.Figure 5 reports no memorization detected for 208/370 monolingual and 119/146 multiway cases after accounting for additional leakage.Source: §A.9 · p. 34, §A.10 · p. 34, Figure 5 · p. 53

A.11 Canaries

  • The canary inventory totals 1,945,631 examples, including 1,279,635 for MADLAD-400 and 585,596 for parallel data, scaled by language size to limit training impact.Source: §A.11 · p. 49

A.11.1 MADLAD-400 Canaries

  • MADLAD-400 canaries use interleaved document chunks, shuffled tokens, or real-text prefixes completed with random tokens; their number and repetition vary by language resources.Source: §A.11.1 · p. 49

A.11.2 Parallel Canaries

  • Parallel canaries combine random or shared-prefix source inputs with interleaved or shuffled targets; 21,120 additional natural-source canaries are limited to language pairs with at least 500,000 examples.Together, the two designs yield 585,596 parallel canaries.Source: §A.11.2 · p. 51

A.11.3 Generic Canaries

  • The 80,400 generic canaries are deliberately unlike natural data and comprise fully random or random-prefix examples with prefixes of 50, 100, or 200 tokens.Source: §A.11.3 · p. 52

Composition

  • MADLAD-400 is a general-purpose, CommonCrawl-derived dataset with 4.0B documents across 419 languages, but aggressive filtering may omit documents or languages and leave mislabeled or low-quality data.The dataset may also contain confidential information because it is web-crawled.Source: Motivation · p. 54, Composition · p. 54

Collection

  • MADLAD-400 was formed by annotating CommonCrawl with a LangID model trained using, then manually inspecting and filtering documents from snapshots available through August 20, 2022.The authors performed the audit, with native-speaker volunteers advising on quality in some cases.Source: Collection · p. 55
  • No ethical review or data-protection impact analysis was conducted, and individuals were not notified or asked to consent to collection.The dataset may include sentences about people, and names may identify individuals.Source: Collection · p. 55, Composition · p. 55

Preprocessing/cleaning/labeling

  • Preprocessing included three-sentence-span deduplication, heuristic quality filtering, script-encoding corrections, a Chinese pornography filter, and removal of 79 low-quality or mislabeled language datasets.Raw data remains hosted by CommonCrawl rather than being saved with the processed dataset.Source: Preprocessing/cleaning/labeling · p. 55

Uses

  • MADLAD-400 has been used for machine translation and language modeling and could serve as general training data for its languages, but users should assess its suitability for their use case.Sensitive content about individuals or groups could affect some downstream NLP tasks.Source: Uses · p. 56

Distribution

  • AI2 hosts the dataset; a version is available under ODC-BY, with CommonCrawl terms applying to its content, and no routine updates are planned.Major reported issues may be corrected, and MADLAD-400-noisy is available for users to build upon with alternative cleaning or preprocessing.Source: Distribution · p. 56, Maintenance · p. 56

Ethical Considerations

  • Despite filtering, MADLAD-400 may contain sensitive or offensive content and identifiable information, which could affect downstream performance or produce problematic outputs.The models are research baselines, not assessed for production use, and are not intended to work out of the box on domain-specific tasks.Source: Composition · p. 55, Uses · p. 56, Ethical Considerations · p. 57

Training Data

  • The models use MADLAD-400 for training; the MT model also uses parallel data covering 157 languages, and evaluation covers only 204 supported languages on MT and few-shot MT tasks.Evaluation uses WMT, NTREX, Flores-200, and Gatones; translation quality varies by language, while domain variation has not been assessed. SacreBLEU and chrF are the reported evaluation metrics.Source: Training Data · p. 57, Caveats and Recommendations · p. 57, Evaluation Data · p. 57

Composition

  • The memorization-canary dataset contains 1,945,631 instances derived from MADLAD-400 sentences, with token modifications that make samples outliers; some canaries are duplicated to study repetition.It is a subsample of MADLAD-400, and the described canaries are included in the release; modifications reduce the chance of exposing sensitive, confidential, or identifying content from the source.Source: Composition · pp. 58–59

Collection

  • A predefined number of sentences was uniformly sampled from each language after random subsampling of MADLAD-400 documents from CommonCrawl snapshots available by August 20, 2022.The data was obtained second-hand; no ethical review or consent was reported, and the authors note the underlying dataset may contain information about people, though canary modifications reduce that chance.Source: Collection · p. 59

Preprocessing/cleaning/labeling

  • Sentences were either interleaved in batches of 20–50 tokens or shuffled at the token level.The raw MADLAD-400 data was saved, but preprocessing software was unavailable as of June 13, 2023.Source: Preprocessing/cleaning/labeling · p. 59

Uses

  • MADLAD-400 has been used for machine translation and could support studies of memorization in broad language-modeling scenarios.Users are urged to consult the MADLAD-400 datasheet to understand risks associated with the canaries; no prohibited tasks were specified.Source: Uses · pp. 59–60

Distribution

  • MADLAD-400 was scheduled for distribution in June 2023 through a GCP bucket; an AI2 version is available under ODC-BY, subject to CommonCrawl terms of use.Users are bound by CommonCrawl terms of use for the content in the dataset.Source: Distribution · p. 60

Maintenance

  • AI2 hosts the dataset, with no planned updates or continued support for older versions; major issues may be corrected when reported by email or through GitHub.Others may build on the dataset by generating canaries from MADLAD-400.Source: Maintenance · p. 60

Related papers

Loading 2309.04662v1…