Source-linked AI summary
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources
Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilić, Nurulaqilla Khamis, Colin Leong, Maraim Masoud, Aitor Soroa, Pedro Ortiz Suarez, Zeerak Talat, Daniel van Strien, Yacine Jernite
TL;DR
Large-scale language-data collection raises concerns about data subjects’ rights and the difficulty of examining collections without sufficient documentation and analysis tools. BigScience built a documentation-first catalogue and gathered metadata on potential sources through public hackathons. The resulting catalogue contains 192 sources, with at least 10% of entries representing each target language except programming languages, across locations around the world.
Problem
Large-scale data collection prioritizes volume, while concerns about data subjects’ rights and insufficient documentation and analysis tools make collections difficult to interrogate.
Source: §1 · p. 1Method
BigScience developed an online catalogue to gather metadata on potential sources for 13 target language groups through organized public hackathons.
Source: §1.1 · p. 2Results
192 data sources were catalogued, with at least 10% of entries representing each target language except programming languages, and entries spanning locations around the world.
Source: §6 · p. 7Takeaways & Limitations
The catalogue remains available to BigScience and other projects using the resources, and the team plans to maintain it as a reference for future dataset and modeling efforts.
Source: §6 · p. 7, §7 · p. 8The catalogue covers only a small fraction of world languages, has a nonuniform distribution across target languages, and lacks some in-depth metadata about data characteristics.
Source: §6.2 · p. 8
Abstract
from arXiv · showhide
In recent years, large-scale data collection efforts have prioritized the amount of data collected in order to improve the modeling capabilities of large language models. This prioritization, however, has resulted in concerns with respect to the rights of data subjects represented in data collections, particularly when considering the difficulty in interrogating these collections due to insufficient documentation and tools for analysis. Mindful of these pitfalls, we present our methodology for a documentation-first, human-centered data collection project as part of the BigScience initiative. We identified a geographically diverse set of target language groups (Arabic, Basque, Chinese, Catalan, English, French, Indic languages, Indonesian, Niger-Congo languages, Portuguese, Spanish, and Vietnamese, as well as programming languages) for which to collect metadata on potential data sources. To structure this effort, we developed our online catalogue as a supporting tool for gathering metadata through organized public hackathons. We present our development process; analyses of the resulting resource metadata, including distributions over languages, regions, and resource types; and our lessons learned in this endeavor.
1. Introduction
The paper addresses representation, consent, transparency, self-determination, and privacy concerns in large-scale language-data collection through a documentation-first, human-centered approach. BigScience defined representativeness geographically and sociolinguistically, then documented potential sources for 13 target language groups before collection.
1. Introduction
- Large-scale language-data collection raises concerns about representation, privacy, consent, and the difficulty of interrogating poorly documented collections.Prior sources such as Wikipedia and Common Crawl have been associated with demographic bias, hate speech, and over-representation of sexually explicit content.Source: §1 · p. 1
- The project frames its documentation-first, human-centered approach around representation, consent, transparency, self-determination, and privacy.It aims to create a dataset for large multilingual language models while emphasizing data subjects’ control over their data.Source: §1 · p. 1
1.1. The BigScience Research Workshop
- Representativeness is defined as the intersection of geographic and sociolinguistic context, with dialects and regions targeted for each language.The 13 target groups are Arabic, Basque, Chinese, Catalan, English, French, Indic languages, Indonesian, Niger-Congo languages, Portuguese, Spanish, Vietnamese, and programming languages.Source: §1.1 · p. 2
- The BigScience catalogue documents potential sources before collection, using an online form to support public, region-focused hackathons.The paper presents the form, initial metadata results, and challenges and limitations from the effort.Source: §1.1 · p. 2
2. Related Work
Language-data catalogs range from organizations distributing licensed resources to technical libraries that provide dataset access and tools. Other catalogs, including Masader, provide metadata and categories without hosting the datasets themselves.
2. Related Work
- Language-data organizations such as the Linguistic Data Consortium distribute licensed resources, mainly to institutions, and use fees to support dataset creation, licensing, storage, and maintenance.The passage also names ELRA, Chinese LDC, LDC for Indian Languages, and CLARIN.Source: §2 · p. 2
- NLP and ML libraries such as NLTK, Hugging Face Datasets, and TensorFlow provide tools for loading datasets and access to catalogs of datasets (Bird et al., 2009; Lhoest et al., 2021; Tensor Flow Authors, 2021).Other catalogs provide information about dataset uses and categories without hosting the data, including Papers with Code and Arabic-language Masader (Alyafeai et al., 2021).Source: §2 · p. 2
3. The Catalogue
The catalogue balances metadata needs across data-sourcing, tooling, and governance work with the effort required of submitters. Its online form adapts questions to resource type and records information on languages, locations, custodians, access, legal considerations, source types, and technical properties.
3. The Catalogue
- The catalogue groups and prioritizes nearly 40 metadata items, balancing working-group needs, submitter effort, and applicability across languages and sources.An iterative process produced an openly accessible form with multiple-choice questions and free-response options.Source: §3 · p. 2
3.1. The Catalogue Submission Form
- The form varies questions by resource type—primary source, processed language dataset, or language organization or advocate—and gathers shared and type-specific metadata.All submissions request languages, locations, and representative, owner, or custodian contact information; primary-source and dataset entries also address availability, licenses, PII, data type, and medium.Source: §3.1 · p. 3
3.1.1. General Information
- The general-information section records the resource name, a unique search identifier, homepage, and a searchable description.The description is intended to display when users search the catalogue.Source: §3.1.1 · p. 3
3.1.2. Languages and Locations
- Submitters can specify all represented languages, including target language groups, more specific Indic and Niger-Congo languages, and other languages using BCP-47.The form is designed to accommodate differing levels of language and location granularity.Source: §3.1.2 · p. 3
3.1.3. Representative, Owner, or Custodian
- The form records custodian identity and contact details to support understanding applicable local legal structures, communication, and permission requests.Submitters may link a custodian to an existing catalogue organization or enter its name, type, location, and contact information.Source: §3.1.3 · p. 3
3.1.4. Availability of the Resource
- For primary sources and existing datasets, the form records access routes, licensing or terms of use, and submitter assessments of whether model training respects data creators’ and custodians’ rights.It also asks submitters to assess the likelihood and types of PII, drawing on HIPAA and GDPR categories and advising a default assumption that the resource contains PII absent a strong reason otherwise.Source: §3.1.4 · pp. 3–4
3.1.5. Primary Source Type
- Primary-source entries describe resource contents, while processed-dataset entries can link to documented primary sources and record their data types and licensing.The passage notes that dataset licenses may be unclear or may directly violate source-material terms.Source: §3.1.5 · p. 4
3.1.6. Media Type, Format, Size, and Processing
- The technical section records medium, format, transcription or processing, and descriptive estimates of data amount using submitter-defined units and magnitude ranges.Submitters can review their completed answers as a JSON dictionary before saving the entry.Source: §3.1.6 · p. 4
4. Community Hackathons
The team used online hackathons organized around language communities and regions to gather catalogue entries. Organizers supported participants, and the project limited demographic data collection while seeking feedback through a post-event survey.
4. Community Hackathons
- Hackathons for African, Asian, Basque, English-speaking, and Spanish-speaking communities ran online from October through December 2021, lasting one to six hours.Partners included Masakhane, Machine Learning Tokyo, and LatinX in AI; BigScience members answered questions and discussed resources and institutions.Source: §4 · p. 4
- The form required a name and email to save an entry, but organizers collected no further participant demographics and instead sent a 10-question post-hackathon survey.Events were announced through BigScience and partner social-media accounts.Source: §4 · p. 4
5. Results
The catalogue records a geographically and linguistically broad set of resources, but coverage is uneven and key metadata about licensing and PII is often absent or uncertain. Hackathon participants contributed resources across varied professional and geographic contexts.
5.1. Hackathon Participation
- Forty-one participants submitted resources, and 11 responded to the follow-up survey; respondents varied in location and career stage, and most added resources for languages they knew well.Six respondents reported adding resources only for languages where they were native or advanced speakers.Source: §5.1 · p. 5
5.2. Gathered Resources
- Across 192 entries, 432 language tags were recorded; English is most frequent, English and Spanish account for about half of target-language records, and Chinese appears less often than French, Spanish, or Vietnamese.Entries can carry multiple language tags.Source: §5.2 · p. 5, Figure 1 · p. 5
- More than half of entries’ primary language locations are in Asia and Europe; English, French, Spanish, and Portuguese also have strong European representation alongside other regional and worldwide entries.The location analysis uses only each entry’s first reported location for the regional distribution.Source: §5.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
- Of 192 resources, 98 (51%) are primary sources, 64 (33%) processed sources, and 30 (16%) organizations; universities and research institutions are the most frequent custodian type at 23.44%.Catalan, Indic, and Vietnamese are the exceptions to the pattern of target-language groups having more primary than secondary sources.Source: §5.2 · p. 6, Table 3 · p. 6, Table 4 · p. 6
- The catalogue records 164 custodian locations, though 28 entries lack one; many resources are hosted in the US and a few European countries.The top 12 custodian locations are listed in Table 5.Source: §5.2 · p. 6, Table 5 · p. 6
- About 37% of catalogue entries have open or public-domain licensing, while another 37% have no licensing properties recorded.Entries may have multiple licensing properties.Source: §5.2 · p. 7, Table 6 · p. 7
- More than half of entries record some PII; 33% have unclear or no PII metadata, while only 13% are recorded as having no PII.These figures reflect metadata recorded during the hackathons.Source: §5.2 · p. 7, Table 7 · p. 7
6. Discussion
The catalogue offers a geographically and institutionally diverse set of resources, but its entries remain concentrated in highly resourced languages and locations. The authors describe participation and metadata-collection challenges, catalogue limitations, and possible improvements for future work.
6. Discussion
- The 192-source catalogue represents at least 10% of each target language except programming languages, and its documentation will remain accessible to BigScience and other projects.Most resources are primary sources; participants’ initial PII estimates suggest PII is significant across the sources.Source: §6 · p. 7
6.1. Challenges in creating the catalogue
- Catalogue entries remain concentrated in highly resourced languages and locations, with many languages represented only once.The authors report that 41 participants fell short of desired numbers, while respondents cited difficulty finding resources and metadata and limitations in the catalogue infrastructure.Source: §6.1 · pp. 7–8
6.2. Limitations of the catalogue
- The catalogue covers only a small fraction of world languages, has a non-uniform target-language distribution, and lacks detailed metadata on data quality and language characteristics.Its loosely structured dialect and location ontology offers flexibility but makes precise distribution analysis harder.Source: §6.2 · p. 8
6.3. Recommendations
- The authors suggest breaking catalogue tasks into smaller volunteer roles and collaborating with custodians to improve metadata access and standardization.They also recommend a machine-readable schema while noting that adoption requires time and coordination; users must follow the underlying data’s licensing and PII requirements.Source: §6.3 · p. 8
7. Conclusion
The paper presents a human-centered metadata collection effort and the resulting catalogue for 13 language groups. It plans to fill missing information, add entries, and maintain the catalogue as a reference for future dataset and modeling work.
7. Conclusion
- The authors will continue maintaining the catalogue, filling missing information and adding entries toward greater representation across languages and regions.They hope the work encourages conscientious documentation before data collections are released, especially at large scale.Source: §7 · p. 8
Catalogue
The catalogue supports entry validation and filtered review through an interactive map and language-proportion chart. Users can inspect resource descriptions returned by filters.
Catalogue
- A second participant can review and edit submitted entries, validate each section, and save the reviewed work.Validated entries are marked and retain access to the original entry and later saved versions.Source: Catalogue · p. 9
- The interactive map shows entry counts at geographic levels such as region or country for data creators or custodians.Color indicates country-level counts, while markers identify regions that can be zoomed in on.Source: Catalogue · p. 9
- Users can filter the map and language-proportion pie chart by properties such as resource, license, or media type, then open returned entry descriptions.Source: Catalogue · p. 9
10. Language Resource References
The references list Henry Kučera and Winthrop Nelson Francis’s 1964 Standard Corpus of Present-Day American English.
10. Language Resource References
- Kučera and Francis’s 1964 reference is titled Standard Corpus of Present-Day American English and was published by Brown University Press.Source: §10 · p. 11