Source-linked AI summary
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, Thien Huu Nguyen
TL;DR
Training data for state-of-the-art LLMs is often undisclosed, while multilingual resources can lack sufficient scale and cleaning. CulturaX combines large web-crawled corpora with extensive language-aware cleaning and deduplication, yielding 6.3 trillion tokens across 167 languages and making the dataset publicly accessible.
Problem
Training datasets for state-of-the-art LLMs are often not fully disclosed, and existing multilingual datasets can lack the quality and scale needed for high-performing multilingual LLM training.
Method
CulturaX combines mC4 and OSCAR and applies language identification, URL-based filtering, metric-based cleaning, document refinement, and MinHashLSH deduplication.
Results
6.3 trillion tokens spanning 167 languages remain after cleaning and deduplication, with 46.48% of initial documents removed.
Takeaways & Limitations
CulturaX provides a publicly accessible, large-scale multilingual dataset intended for training LLMs and supporting multilingual learning research.
Takeaways & Limitations
The dataset relies on web-scraped sources because high-quality curated data may not be available for various languages.
Abstract
from arXiv · showhide
The driving factors behind the development of large language models (LLMs) with impressive learning capabilities are their colossal model sizes and extensive training datasets. Along with the progress in natural language processing, LLMs have been frequently made accessible to the public to foster deeper investigation and applications. However, when it comes to training datasets for these LLMs, especially the recent state-of-the-art models, they are often not fully disclosed. Creating training data for high-performing LLMs involves extensive cleaning and deduplication to ensure the necessary level of quality. The lack of transparency for training data has thus hampered research on attributing and addressing hallucination and bias issues in LLMs, hindering replication efforts and further advancements in the community. These challenges become even more pronounced in multilingual learning scenarios, where the available multilingual text datasets are often inadequately collected and cleaned. Consequently, there is a lack of open-source and readily usable dataset to effectively train LLMs in multiple languages. To overcome this issue, we present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages, tailored for LLM development. Our dataset undergoes meticulous cleaning and deduplication through a rigorous pipeline of multiple stages to accomplish the best quality for model training, including language identification, URL-based filtering, metric-based cleaning, document refinement, and data deduplication. CulturaX is fully released to the public in HuggingFace to facilitate research and advancements in multilingual LLMs: https://huggingface.co/datasets/uonlp/CulturaX.
1 Introduction
CulturaX addresses limited transparency, scale, and quality in multilingual LLM training data by releasing a large, cleaned, and deduplicated dataset. It targets training across 167 languages with substantial non-English coverage.
- Training datasets for many publicly available state-of-the-art LLMs remain incompletely disclosed, limiting analysis and replication.The paper links this lack of transparency to difficulties studying attribution and fundamental LLM issues.
- Existing open-source multilingual datasets often fall short in quality or scale for training high-performing LLMs.The paper specifically notes that multilingual datasets are inadequately collected and cleaned, while OSCAR lacks document-level deduplication.
- CulturaX combines mC4 and OSCAR into a multilingual dataset with 27 TB of text and 6.3 trillion tokens across 167 languages.More than half of the dataset is dedicated to non-English languages.
- CulturaX is released publicly to support research and development in multilingual learning.
- The dataset applies extensive cleaning and document-level deduplication to remove noisy, non-linguistic, toxic, incorrectly identified, and redundant content.The pipeline uses metric thresholds, language identification, and MinHashLSH and URL-based deduplication or filtering.
2 Multilingual Dataset Creation
CulturaX combines mC4 and OSCAR and processes their web-crawled text through extensive cleaning and deduplication. Its pipeline uses language, URL, metric, document, and similarity-based filtering with language-specific thresholds.
- 2 Multilingual Dataset Creation: CulturaX combines mC4 and OSCAR, two large multilingual datasets, then applies cleaning and deduplication to produce training data for multilingual LLMs.The source corpora are extracted from CommonCrawl and cover many languages.
- 2 Multilingual Dataset Creation: The cleaning procedure filters language-identification errors, harmful URLs, noisy metrics, and poorly formed documents.The stated stages include language identification, URL-based filtering, metric-based cleaning, and document refinement.
- 2 Multilingual Dataset Creation: FastText language-identification confidence is used to support filtering, addressing the weaker cld3 identification associated with mC4.
- 2.1 Data Cleaning: The metric pipeline uses document-level values such as word counts, repetition ratios, stopword ratios, language confidence, perplexity, and document length to remove outliers.Documents in abnormal metric ranges are classified as outliers or noise and removed.
- 2.2 Data Deduplication: MinHashLSH filters similar documents within each language using MinHash signatures, n-grams, hash functions, and Jaccard similarity.
3 Data Analysis and Experiments
After cleaning and deduplication, CulturaX contains 6.3 trillion tokens across 167 languages. The pipeline removes a substantial portion of the initial documents while retaining the final multilingual corpus.
- 6.3 trillion tokens remain across 167 languages after all cleaning and deduplication steps.
- 46.48% of the initial documents are removed by the data-cleaning pipeline.The paper presents this reduction as evidence that the approaches filter noisy information from the multilingual datasets.
4 Related Work
Related work distinguishes curated and web-crawled training data and highlights limited transparency and non-English coverage in existing resources. CulturaX is positioned as an open, large-scale, cleaned multilingual alternative.
- Curated data comes from targeted sources, whereas web-crawl data gathers varied text from across the internet and supports much larger collections.CommonCrawl is described as a large web-crawl repository used by recent LLMs.
- Training datasets for recent state-of-the-art generative LLMs are often not fully released, including datasets for some models described as open-source.
- Existing open-source datasets remain predominantly English-focused and provide limited data for other languages.
- Existing multilingual datasets do not consistently meet the quality and scale requirements for effectively training generative LLMs.The paper identifies missing document-level fuzzy deduplication, weaker mC4 language identification, limited BigScience ROOTS samples, and outdated CC100 information as examples.
- CulturaX offers multilingual, open-source, large-scale data that is cleaned and deduplicated for readily usable LLM training.
5 Conclusion
CulturaX provides a cleaned and deduplicated multilingual dataset spanning 167 languages, with 6.3 trillion tokens, openly accessible for training and research.
- 6.3 trillion tokens across 167 languages make CulturaX a large-scale multilingual training dataset.
- A comprehensive cleaning and deduplication pipeline produces a high-quality dataset for training high-performing multilingual LLMs.
- CulturaX is openly accessible to promote further research and applications in multilingual learning.