Source-linked AI summary
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
TL;DR
Low-resource multilingual web datasets are widely available, but their quality and usability are insufficiently known because direct evaluation is limited. The paper manually audits 230 per-language subsets from five major datasets, finding low valid-data rates, systematic quality problems, and labeling errors. It also analyzes these issues automatically and recommends low-effort auditing and improved release practices.
Problem
Low-resource languages are increasingly covered by multilingual crawl releases, yet their data quality and usability remain unknown because direct evaluation research is scarce.
Method
The paper manually audits 230 per-language subsets from five major crawled multilingual datasets, proposes low-effort auditing techniques, and supplements the audit with automatic analyses.
Results
15 of 205 audited language-specific samples contained no single correct sentence, while audited quality varied substantially and lower-resource corpora tended to have lower quality.
Takeaways & Limitations
Multilingual corpus releases should use dedicated quality evaluation, auditing techniques, and clearer language labeling rather than relying primarily on downstream improvements.
Takeaways & Limitations
The analyses omit some issue classes, including close dialects and the estimated share of machine-generated content, and note that suitable automatic metrics require more research.
Abstract
from arXiv · showhide
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.
1 Introduction
Web-derived multilingual datasets have expanded access to text for hundreds of languages and enabled multilingual models, but their quality is rarely measured directly. This gap is especially consequential for low-resource languages, whose data quality and usability remain uncertain.
- Web-derived collections now provide downloadable text for hundreds of languages and support highly multilingual models.
- Dataset quality is rarely measured directly and is instead often judged through downstream application improvements.
- Low-resource languages face data scarcity, while research rarely evaluates their data collections or crawling and filtering tools.
- The study manually audits 230 per-language subsets from five major crawled multilingual datasets and proposes low-effort auditing solutions.
- Its analyses identify low amounts of valid in-language data, systematic dataset and language issues, and nontransparent or incorrect language codes.
2 Related Work
Prior work documents noisy web crawls, filtering methods, dataset documentation, and selected language-identification problems. However, systematic quality evaluation of multilingual, low-resource web datasets remains limited.
- Web-crawled corpora are known to be noisy, and lower-resource languages have especially serious segment-level language-identification issues.
- Cleaning and filtering web crawls can improve general language modeling and downstream task performance.
- Dataset documentation frameworks seek to establish minimum standards for describing NLP datasets and related data-science resources.
- The scale of machine-learning research makes automatically collected and curated datasets increasingly difficult to validate.
- Closest prior analyses examined OSCAR language content or C4 sources, whereas this work audits five major multilingual datasets and broader quality issues.
3 Multilingual Corpora
The paper compares five web-derived multilingual corpora spanning parallel and monolingual data, with differing sources, language-identification systems, filtering procedures, and evaluation contexts.
- Except for WikiMatrix and ParaCrawl, the corpora are derived from CommonCrawl.
- CCAligned: CCAligned builds parallel data from 68 CommonCrawl snapshots by aligning same-URL documents with differing language codes, then refining alignments with LASER.
- Table 1 compares parallel and monolingual web corpora and their downstream evaluation tasks, including BLEU-based machine translation and POS/DEP evaluation.
- mC4: mC4 is a monolingual dataset in 101 languages from 71 CommonCrawl snapshots, filtering short or bad-word-containing pages before sentence splitting and deduplication.
- OSCAR: OSCAR extracts deduplicated plain-text WET data from CommonCrawl and applies FastText language identification at the line level without other filtering.
- ParaCrawl: ParaCrawl v7.1 covers 41 mainly English-centered language pairs and uses Bitextor for crawling, alignment, normalization, and Bicleaner filtering.
- WikiMatrix: WikiMatrix contains 135M parallel sentences in 1620 language pairs across 85 languages, mined from Wikipedia with FastText LangID and LASER.
4 Auditing Data Quality
The audit samples sentence-level quality across selected languages and datasets, revealing severe low-resource data problems, systematic labeling errors, and important limits on interpretation.
- Dataset Quality: 15 of 205 language-specific samples (7.3%) contained no correct sentence, while CCAligned had 44 of 65 and WikiMatrix 19 of 20 audited languages below 50% correct.Across datasets, macro-averaged correctness ranged from 24% to 87%.
- Aggregation Bias: Micro-averaging gave a more optimistic picture because high-resource languages were upweighted; in mC4, the eight largest languages accounted for over 63% of random samples and had near-perfect quality.This imbalance can allow poor quality in underrepresented languages to go unnoticed without dedicated evaluation or proficient speakers.
- Error Types: Non-linguistic content was more common than wrong-language content; CCAligned averaged 31.42% non-linguistic and 9.44% wrong-language content, while mC4 averaged 11.40% and 15.98%, respectively.Four of 48 audited mC4 languages had more than 50% content in other languages.
- Resource Level: Quality correlated positively with dataset size in every dataset, strongest for mC4 (r = 0.66), but high-resource languages could still be severely corrupted and mid-resource results were mixed.Examples include Javanese at 5%C in CCAligned and a romanized Urdu corpus at 0.5% C versus 100%C for Urdu.
- Annotation Quality: Non-proficient speakers could support low-effort auditing, but finer-grained label accuracy dropped and the taxonomy omitted unnatural in-language text and had ambiguities across languages.The authors specifically note problems with correct-short and correct-boilerplate labels, including for agglutinative languages such as Turkish.
5 Dataset Mis-labeling
The audit finds widespread language-code inconsistencies, including nonstandard codes, serious mislabelings, deprecated identifiers, and ambiguous supersets. Across the analyzed datasets, it reports 83 nonstandard codes, excluding 59 codes affected by superset issues.
- 83 nonstandard language codes were identified across CCAligned, OSCAR, mC4, WikiMatrix, and JW300, excluding 59 codes affected by superset issues.The counts are 8 in CCAligned, 3 in OSCAR, 1 in mC4, 1 in WikiMatrix, and 70 in JW300.
- Some datasets shorten or invent codes, while others assign labels for one language to text in another.Examples include CCAligned's zza → zz and shn → qa, and OSCAR's als-labeled data that is actually gsw.
- 12% (48/417) of JW300 entries carrying sign-language codes contain texts in another high-resource language rather than sign-language transcripts.The en-zsl corpus is English-English parallel data consisting of copies.
- Superset codes make the particular language difficult to determine, although some conventions, such as Arabic ar, may not create a practical problem.WikiMatrix includes Serbian, Croatian, Bosnian, and the superset Serbo-Croatian code sh; many superset-coded data remain mysterious.
- Deprecated codes are used in WikiMatrix, mC4, OSCAR, and JW300.The reported examples are sh, iw, sh and eml, and daf, respectively.
6 Risks of Low-Quality Data
Low-quality multilingual data can distort both model evaluation and downstream use. Misaligned or mislabeled content may produce plausible but factually wrong outputs, while benchmark artifacts can make progress appear greater or harder than it is.
- Low-quality corpora can propagate failures through translation-based pipelines into downstream applications without revealing their causes.The paper gives question answering and summarization as examples of downstream applications built from translated data.
- Low-quality benchmarks may exaggerate model performance or make low-resource tasks appear harder or infeasible.These effects can redirect productive effort away from affected tasks and languages.
- Translation-quality correlations with audit quality varied by direction, with reported Spearman values of ρ = 0.32, p = 0.14; ρ = 0.74, p = 0.000078; and ρ = 0.80, p = 0.0000087.
- Parallel-looking but factually incorrect translations may lead models to produce plausible false translations that users trust.The paper identifies algorithmic trust and automation bias as factors that can amplify this risk.
7 Future Work and Recommendations
The audit finds widespread quality problems and recommends lightweight inspection before dataset use or release, followed by targeted cleaning or documentation. It also calls for broader evaluations and better automatic metrics while noting important unmeasured content sources.
- 87 of 205 audited languages had under 50% usable data, including 15 languages with 0% in-language content.
- Future work should improve human evaluation, develop standard automatic dataset metrics, and estimate machine-generated content from MT, language models, bots, or templates.The paper notes that machine-generated text may remain useful while introducing artifacts or overrepresenting generation patterns.
- A quick scan of 100 sentences can detect major problems without requiring evaluator proficiency in the language.Annotating a small sample can also reveal actionable filtering or usage strategies.
- Recommended cleaning methods include length-ratio, LangID, TF-IDF wordlists, dictionaries, and neural language-model scoring.The paper cautions that no method provides a quick, easy fix, especially for low-resource languages.
- When noisy data contains desirable content, documentation such as per-language quality scores, datasheets, or nutrition labels can complement filtering.The authors discourage releasing corpora with near-zero in-language content because they may appear usable.
- The community is encouraged to continue auditing and evaluating public datasets in ways analogous to system-comparison studies.
A Details on Language Code Issues
The appendix documents extensive JW300 language-code problems, including sign-language mislabeling, malformed private-use extensions, ambiguous equivalencies, and standards violations. It also lists miscellaneous coding errors in CCAligned and OSCAR.
- Tables 7–10 provide complete lists of code equivalencies, sign-language mislabels, JW300 variety issues, and miscellaneous CCAligned and OSCAR errors.
- Twelve JW300 codes beginning with jw_ are mis-parsed private-use extensions rather than varieties of Javanese.They should have been represented using BCP-47 extensions in the form <supercode>_x_<tag>.
- 48 of 417 JW300 entries labeled as sign languages actually contain unrelated high-resource-language text, usually English or Spanish.The table identifies the actual language associated with each sign-language code.
- JW300 also contains three codes equivalent to ISO codes, one deprecated ISO code, and four ISO639-3 codes that are not BCP-47.
- The codes hy_arevmda and rmy_AR misuse private subtags and represent Western Armenian and Kalderash Romany, respectively.
- Three code pairs—nya/ny, qu/que, and kmr/kmr_latn—create ambiguity despite representing equivalent language or script distinctions.The paper states that private-use extensions would have been appropriate in these cases.
B Complete Error Taxonomy and Instructions
The audit taxonomy labels sentence-level quality and language problems across parallel data, distinguishing correct content from short, boilerplate, incorrect, wrong-language, non-language, and unresolved cases. An auxiliary unknown label marks items requiring verification.
- CC denotes a correct, natural translation, including sufficiently long sentence fragments.The guideline says fragments are acceptable when they are not too short, at about five words or greater.
- CS denotes correct translations that are single words, short phrases, or highly repeated short phrases.
- CB denotes correct but boilerplate content that is autogenerated, formulaic, or generally not useful to NLP models.The instructions acknowledge that boilerplate is often difficult to identify consistently.
- X marks parallel sentences whose source and target are in the correct languages but are not adequate translations.
- WL marks wrong-language content, while NL marks cases where at least one side is not linguistic content.Proper-noun-only sentences are classified as NL under the instructions.
- U is an auxiliary unknown label for sentences requiring native-speaker verification and is resolved in most cases.
C Methodological Notes
The audit can identify many data-quality problems through simple inspection, web searches, and basic research rather than language expertise. Repeated errors and metadata clues make systematic problems easier to detect.
- No Skills Required: Searching a sentence online often reveals its source page, language clues, or a translated counterpart.Page URLs, headlines, and autonyms can help identify the language when direct inspection is insufficient.
- No Skills Required: Anyone who can read the script can usually identify sentences written in the wrong language, despite tricky proper nouns.The audit treats wrong-script text as wrong language, while accounting for languages with published script variants such as bg-Latn.
- No Skills Required: Different numbers in purportedly parallel source and target sentences are an easy signal of unusable autogenerated content.This check does not require knowing either language.
- No Skills Required: Repeated mistranslations can be spotted across a corpus because the same error often recurs many times.Error repetition allows a local observation to reveal a broader corpus pattern.
- Basic Research Required: Basic research can use inherent textual markers, such as enumerators or verse numbers, to assess sentence content.When other checks fail, searching complete sentences or n-grams on the internet provides another low-effort approach.
D Complete Audit Results
The complete audit results report annotation percentages, corpus sizes, and average sentence lengths for five multilingual datasets. They preserve the datasets’ original language codes while flagging languages with very low correctness.
- Audit Results: Five tables report complete annotation percentages for CCAligned, WikiMatrix, ParaCrawl, mC4, and OSCAR.For each language or language pair, the reported ratio is based on up to 100 sentences assigned labels by the primary annotator.
- Audit Results: The audit records each dataset’s sentence count for every language and the average sentence length of audited sentences.These quantities illustrate differences across languages and contextualize the annotation percentages.
- Annotation Scheme: The C column aggregates all correct sub-codes: CC, CS, and CB.Repeated annotations used for agreement measurement are excluded from the reported ratios.
- Audit Results: Languages with less than 20% correct sentences are highlighted, while original published language codes are retained.The paper cautions that these codes should be handled carefully in future work.