PaperlayerView exact revision on arXiv

Source-linked AI summary

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, Yacine Jernite

arXiv:2303.03915v1cs.CLcs.AI

TL;DR

Large multilingual language-model training requires documented, high-quality text resources, but corpus construction and processing at scale remain difficult to inspect and reproduce. This paper documents the collaborative construction and curation of ROOTS, releases a large gated subset and its tools, and analyzes the resulting multilingual corpus and component diversity. ROOTS spans 59 languages and was shaped by community selection, OSCAR filtering, visualization, and governance considerations.

  • Problem

    Large language models require large, high-quality multilingual datasets, while their construction and processing need better documentation and reusable tooling.

  • Method

    The paper documents collaborative ROOTS sourcing, curation, filtering, analysis, and governance, and releases a large gated subset with data-processing tools.

  • Results

    ROOTS comprises 46 natural languages across 3 macroareas and 9 language families, with English as its largest component at 30.03%.

  • Takeaways & Limitations

    The released corpus and tooling support more documented use of multilingual training data and further research on large-scale language-model corpora.

  • Takeaways & Limitations

    Crawled data contains machine-generated content, pornographic spam, and personal information that require mitigation.

Abstract

from arXiv · show

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

1 Introduction

BigScience organized an international, value-driven effort to build and document a representative multilingual dataset for large language modeling. The paper presents a gated ROOTS release together with the tools and analyses developed for its curation.

  • BigScience began in May 2021 as a one-year open collaboration focused on studying and training large language models.The initiative gathered over a thousand participants worldwide.
  • The workshop aimed to train an open-access, massively multilingual LLM using a better documented and more representative multilingual dataset.The target model was intended to be comparable in scale to GPT-3.
  • Four working groups coordinated data governance, sourcing and preparation, privacy, and legal scholarship.These groups oversaw data collection, curation, privacy-risk mitigation, and the international legal context of the workshop.
  • The paper releases a preliminary gated subset of ROOTS and tools for curating, sourcing, cleaning, and inspecting its 498 constituent datasets.The release is subject to committing to the BigScience ethical charter.
  • ROOTS covers 59 languages, including 46 natural languages and 13 programming languages, using community-selected sources and filtered OSCAR web-crawl text.The two components contribute 62% and 38% of the corpus, respectively.
  • The project builds on work emphasizing large text corpora, reusable processing pipelines, and detailed documentation of training data.Prior documentation efforts analyze corpus composition, properties, social impacts, licensing, and consent.

2 (Crowd) Sourcing a Language Resource Catalogue

BigScience assembled identified language resources and pseudo-crawled websites through collaborative collection, normalization, filtering, and deduplication pipelines. Visualization tools supported dataset-specific processing decisions by exposing document- and dataset-level effects.

  • Catalogue sourcing: The catalogue effort gathered language resources through open metadata submissions and hackathons with locally focused ML and NLP communities.The effort included collaborations with Masakhane, Machine Learning Tokyo, and LatinX in AI.
  • Catalogue sourcing: Identified sources were collected in an accessible location and mapped to a common format for subsequent processing.The two-phase approach addressed heterogeneous sources, custodians, hosting methods, and formats.
  • Pseudo-crawled data: The pseudo-crawling pipeline retrieved target-domain pages from 18 Common Crawl snapshots and produced text datasets for 539 domains.Pages were fetched from archived WARC records using metadata from the Common Crawl index.
  • Pseudo-crawled data: HTML extraction minified pages, removed selected DOM subtrees, and reconstructed text structure from the remaining nodes.The procedure removed content from tags and low-text-content subtrees before extraction.
  • Code data: GitHub code was collected from BigQuery, deduplicated, and filtered using source-file character, alphabetic-character, line-length, and token-variation criteria.A preprocessing bug additionally restricted the dataset to GPL-licensed files.
  • Quality processing: Manual inspection, cross-pipeline deduplication, removal of unsuitable datasets, and menu or repetitive-content filtering refined the assembled sources.Processing functions were categorized as document- or dataset-scoped and as cleaning or filtering operations.
  • Quality processing: A Streamlit visualization tool displayed how each processing function altered or removed documents and affected estimated dataset-level metrics.The feedback loop enabled iterative updates to processing pipelines.

3 Processing OSCAR

BigScience supplemented community-selected resources with Common Crawl text while addressing crawled-data quality, spam, and privacy risks through language-specific filtering and deduplication. The resulting pipeline used indicators, native-speaker thresholds, and near-duplicate detection to curate OSCAR.

  • Rationale and risks: Common Crawl data was added to support comparability with prior LLM training artifacts and because recent work extensively used this source.The paper treats OSCAR as a complementary corpus component.
  • Rationale and risks: Crawled data presents risks from machine-generated content, pornographic spam, and personal information.The processing section explicitly frames these as issues requiring mitigation.
  • Quality filtering: Filtering removed documents with excessive repetition, special-character ratios, insufficient closed-class-word ratios, flagged-word ratios, perplexity, or word counts.These indicators target repetitive content, crawling artifacts, SEO, pornographic spam, non-natural language, and insufficient context.
  • Quality filtering: Native speakers established language-specific thresholds for Arabic, Basque, Bengali, Catalan, Chinese, English, French, Hindi, Indonesian, Portuguese, Spanish, Urdu, and Vietnamese.The filtering code, parameters, and cutoff values were publicly released.
  • Filtering analysis: Table 1 reports the percentage of documents removed after all filters for each language, while Figure 3 reports independent discard percentages by filter.The table uses ISO 639-1 language codes.
  • Deduplication: Near-duplicate OSCAR documents were targeted with SimHash after exact deduplication had already been performed.SimHash associates similar texts with hashes having low Hamming distance.
  • Privacy processing: The redaction rules targeted identifiers, email addresses, social-media handles, and IPv4 or IPv6 addresses.Regular expressions skipped years and simple numbers when redacting numeric or alphanumeric identifiers.

4 A First look at ROOTS

ROOTS combines 1.6 terabytes of multilingual text with analyses of its language composition, document sizes, filtering distributions, code subset, and tokenizer-based corpus similarity. These analyses expose both the corpus's broad coverage and component-level differences across languages and source domains.

  • Corpus composition and size: 1.6 terabytes of multilingual text constitute ROOTS, whose size is contextualized against corpora used to train large language models.The corpus is analyzed through aggregated statistics and comparisons with other training corpora.
  • Corpus composition and size: 46 natural languages span 3 macroareas and 9 language families, with English comprising 30.03% of the corpus.Simplified Chinese accounts for 16.16%, French 12.9%, Spanish 10.85%, Portuguese 4.91%, and Arabic 4.6%.
  • Document and filter distributions: 1,129 bytes is the median document size, while Figure 5 reports per-language distributions using logarithmic-scale box-and-whisker plots.The plots show medians, first and third quartiles, 1.5-IQR whiskers, and outliers.
  • Document and filter distributions: Filter-value distributions reflect diverse sourcing and filtering, including a right-skewed flagged-word distribution for OSCAR relative to the Catalogue.The comparison remains skewed for OSCAR even after filtering.
  • Programming languages: 13 programming languages comprise the code subset, with Java, PHP, and C++ accounting for more than half of all documents.The corpus also applies heuristics to identify configuration and test files and MinHash-LSH to detect near-duplicate code.
  • Tokenizer analysis: BLOOM and GPT-NeoX produce a very similar ordering of English components by tokens per byte, whereas T5 shows bumps for domains absent from C4.Tokenizer analysis also surfaces components that resemble ROOTS more than the Pile or C4, including technical, academic, news, and specialized forum data.
  • Tokenizer analysis: High-fertility outlier components tend to be shared across Simplified Chinese, French, and Arabic, reflecting formatting, multilingual definitions, technical terms, forums, or dialectal variation.The authors deem most deviations acceptable as representations of diverse text uses.

5 Conclusion

The paper presents ROOTS as a massive multilingual corpus built through international, multidisciplinary, value-driven collaboration. It also releases project tooling and pursues a release strategy informed by licensing and governance needs.

  • ROOTS is a massive multilingual corpus created through international collaboration among multidisciplinary researchers studying large language models.The corpus assembly followed a value-driven, data-first approach to training BLOOM.
  • The project releases the tooling developed to curate, source, clean, and inspect the corpus.The release strategy is being implemented with licensing and governance needs considered for each data source.
  • The authors hope the work supports more reflective use of data in large language models.

Ethical Considerations and Broader Impacts Statement

BigScience made ethical reflection, social impact, governance, privacy, diversity, and inclusivity central to constructing and releasing ROOTS. Its value-driven process linked collective deliberation to technical and data decisions while recognizing unresolved challenges.

  • Values and governance: The project treated data release, governance, human data selection, web crawling, and privacy tooling as both moral exercises and technical contributions.The authors state that each area requires further research and progress.
  • Broader impacts: Ethical deliberation connected core values with social and legal considerations affecting people directly and indirectly impacted by language-model technologies.The paper presents consensus-seeking and interdisciplinary discussion as important for defining the object of analysis and addressing social impact.
  • Broader impacts: BigScience responded to concerns that NLP and machine-learning research often prioritizes performance and efficiency over care and social impact.The paper situates its approach within prior critiques of narrow evaluation priorities and data practices.
  • Values and governance: BigScience’s ethical charter formalized openness, reproducibility, responsibility, diversity, and inclusivity as shared bases for project decisions.The charter was intended to guide individual and collective commitments and support text curation representing diverse languages, varieties, and uses.

Putting Our Values into Practice

BigScience put its values into practice through participatory, multilingual curation, governance work, and technical tools addressing web-scale data risks. The approach aimed to improve relevance, expertise, documentation, trust, and collective ownership, but retained significant privacy, consent, and representation limitations.

  • Participation: Participatory curation involved communities throughout project design, data curation, model training, and release rather than limiting participation to post hoc evaluation.The paper contrasts this engagement model with prior large-model development conducted largely in-house.
  • Participation: Language-community involvement was intended to reduce divergences between creators’ intentions and multilingual resources produced without language-specific expertise.The authors also argue that collaboration can make artifacts more relevant to participating communities.
  • Participation: Community efforts distributed the human work required for intentional curation and documentation while supporting trust and collective ownership of artifacts.The paper presents these as practical reasons for involving many participants in community-based data work.
  • Legal and governance practice: BigScience addressed web-scraping’s unclear legal status through consultation between its Legal Scholarship and Data Governance working groups.The paper notes that fair use protections vary across jurisdictions and may not cover industry research.
  • Limitations: The approach remains limited by Common Crawl’s tension with consent and privacy, incomplete personally identifiable information removal, and difficulty obtaining contributors’ consent.Similar concerns apply to other catalogue datasets, including the WuDao web-based corpus.
  • Limitations: Reliance on medium-to-large digitized sources can over-represent privilege despite intentional selection for geographic and domain diversity.The supplied passage identifies representativeness as an unresolved limitation of the corpus.

B Details on tools used to obtain crowdsourced dataset

The curation pipeline reconstructs textual structure from HTML by using tag-type heuristics and traversing the HTML DOM tree. This approach approximates rendered structure without reproducing every browser-specific rendering condition.

  • HTML structure: HTML tag types provide structural cues that the pipeline uses to reconstruct the organization of extracted text.The paper notes that browser rendering and CSS can vary substantially across situations.
  • HTML structure: The HTML DOM is represented as a tree and traversed as part of reconstructing text structure.The method uses the DOM rather than relying solely on the original HTML code sequence.

B.2 Visualization tool use cases

The visualization tool supported iterative development of cleaning and filtering pipelines by showing document-level changes and dataset-level effects. Examples illustrate how rendering and repeated templates informed processing decisions.

  • Pipeline development: The visualization tool showed how processing functions altered or removed documents and changed estimated dataset-level quantities.This feedback was used to adapt pipelines and introduce functions for specific cases.
  • HTML rendering: Figure 9 contrasts block-level and inline HTML rendering: <p> creates separated blocks, whereas <cite> remains on the surrounding text line.The example explains why tag types provide useful heuristics for reconstructing text structure.
  • HTML DOM: Figure 10 maps an HTML snippet to its corresponding DOM tree, with yellow squares marking nodes containing textual content.The figure provides a visual model for the tree traversed during text reconstruction.
  • Template removal: For pseudo-crawls, visualization exposed repeated template lines across documents and helped assess how much content the initial removal pipeline discarded.The tool therefore supported refinement of template-removal methods through dataset-size estimates and inspection.

B.3 Exhaustive list of functions used in (Crowd)Sourced dataset

The crowdsourced dataset pipeline uses ordered cleaning, filtering, sentence-processing, and deduplication functions tailored to heterogeneous sources. It also supports iterative inspection and revision of processing choices.

  • Text extraction: HTML text is concatenated according to document structure before unwanted HTML and web artifacts are removed.The pseudo-code separates block-type and inline-type elements while constructing text from a depth-first traversal.
  • Pipeline revision: An iterative visualization workflow exposes document changes and dataset-level effects so users can revise processing pipelines.One revised pipeline expanded the generated dataset from 7Mb to 134Mb, an x18 increase, after preserving data removed by the first iteration.
  • Pipeline organization: The pipeline applies dataset-specific sequences of cleaning and filtering functions to improve extracted text quality.These functions address artifacts such as HTML spans, empty or short documents, metadata types, and repeated content.
  • Language processing: Language-aware processing includes reference removal, sentence splitting, and tokenization choices for multiple language groups.The sentence splitters use Stanza, NLTK, Indic NLP, and other language-specific tools.
  • Deduplication: The pipeline removes duplicate documents using normalized text or URL-based matching, while also applying source-specific deduplication rules.URL normalization can ignore query parameters and treat variants such as a URL and its /amp version as equivalent.

C Exhaustive list of human curated filters used on OSCAR

Human-curated OSCAR filters combine document cleaning, language- and content-quality indicators, repetition measures, and language-specific tokenization. The filters are designed to remove non-natural, low-quality, repetitive, or excessively flagged documents while limiting losses for low-resource languages.

  • Cleaning and quality filters: Cleaning standardizes whitespace and removes links, non-printable characters, and excessively long words to eliminate non-natural document content.The targeted material includes machine-generated or non-language text such as URLs.
  • Language-specific processing: Chinese and Vietnamese receive specialized tokenization because their word boundaries differ from ordinary whitespace segmentation.Chinese uses a SentencePiece unigram tokenizer, while Vietnamese adds two- and three-token sequences to space-separated tokens.
  • Repetition filters: The character repetition ratio uses the k most frequent character n-grams relative to all n-gram occurrences, with k = ⌊N⌋ balancing short-text and long-text effects in practice.Here, N is the number of distinct n-grams; the filter discards documents whose ratio exceeds a language-determined threshold.
  • Repetition filters: The word repetition filter complements character-level filtering by targeting repeated long sentences without the same systematic length bias.Character repetition filtering targets short- to medium-sized repetitions, whereas word repetition filtering is more robust for long exact duplicated sentences.
  • Cleaning and quality filters: The OSCAR pipeline filters documents using special-character, closed-class-word, flagged-word, language-identification, perplexity, and word-count indicators.Native speakers reviewed language-specific closed-class and flagged-word lists, while language identification uses fastText and perplexity uses SentencePiece and KenLM models.
  • Cleaning and quality filters: Approximately 1% of documents for each language are removed by the flagged-word filter to limit pornographic over-representation.The flagged-word lists are restricted to words systematically associated with pornography and exclude medical, scientific, everyday, racial, sexual-orientation, and insulting terms.

D PII filtering initiative

BigScience released Muliwai, a multilingual preprocessing library for anonymization and synthesis, but did not operationalize its transformer-based PII approach on ROOTS. The project instead used a simpler approach because of scale, assessment, and compute-time constraints.

  • Muliwai: Muliwai supports preprocessing, augmentation, anonymization, and synthesis across more than 100 languages using transformer models and backtranslation.Its models include XLM-Roberta and M2M100, with language-specific or cross-lingual alternatives.
  • PII detection: The library detects personal information such as ages, emails, dates, times, addresses, phone numbers, and government-issued identifiers using regexes and context.Some regex matches use surrounding text to improve precision.
  • Deployment constraint: Muliwai was not operationalized on ROOTS because the data scale, unknown language-modeling impact, and compute-allocation time limit prevented deployment.The authors state that they consequently fell back to a simpler approach.

E Data Sources

The ROOTS data sources comprise a broad catalogue of multilingual corpora, web-derived collections, parallel datasets, and language-specific resources. The section also records dataset inventories, corpus linguistic makeup, pseudocrawled data, and contributor roles.

  • Language-specific resources: The catalogue includes Arabic, Bengali, Chinese, Catalan, Hindi, Indonesian, Swahili, Vietnamese, and many other language-specific datasets.Examples include AraBench, BanglaLM, DuReader, IndicNLP Corpus, Indonesian News Corpus, and Vietnamese corpora.
  • Multilingual and parallel resources: Multilingual and parallel resources include OPUS-100, Samanantar, WIT3, MultiUN, Opensubtitles2016, and the United Nations Parallel Corpus.These resources span combinations of European, Asian, African, and other languages, including code and parallel text.
  • Web and curated sources: The source collection also contains encyclopedic, literary, educational, news, conversational, and web-scale material from Wikipedia, Wikisource, Project Gutenberg, Books by Book Dash, and Common Crawl.Additional sources include Wikiquote, Wikiversity, Wikivoyage, Wiktionary, Wikinews, and the Pile.
  • Corpus inventories: Tables summarize the crowdsourced datasets, the corpus’s linguistic makeup, and pseudocrawled data by language and number of domains.These are presented as Tables 2, 3, and 4.
  • Curation and contributions: Contributors gathered sources, integrated datasets, implemented loading and filtering tools, performed deduplication and PII filtering, and supported language-specific curation.The listed roles also include analysis, visualization, licensing outreach, and paper writing.
Loading 2303.03915v1…