Source-linked AI summary
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
TL;DR
Instruction-tuning resources remain concentrated in English and a few data-rich languages, leaving many languages underrepresented. Aya responds by combining fluent-speaker curation, dataset templating, and translation into four open resources, including a 204,114-example dataset and a 513M-instance collection. The project broadens multilingual resource coverage while documenting that substantial linguistic gaps remain.
Problem
Instruction fine-tuning datasets have focused almost entirely on English, while many languages remain low-resource or underrepresented.
Method
Aya combines participatory human curation by fluent speakers with templating and translation of existing datasets, releasing four open resources.
Results
The project produces 204,114 human-curated annotations in 65 languages and a 513M-instance collection covering 114 languages.
Takeaways & Limitations
Aya provides open multilingual instruction-tuning and evaluation resources built through collaboration across 119 countries.
Takeaways & Limitations
The resources cover only a small fraction of linguistic diversity, with 93% of the world’s languages still not used to train LLMs.
Abstract
from arXiv · showhide
Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.
1 Introduction
Aya addresses the concentration of instruction-tuning resources in English and a few data-rich languages by building open multilingual resources through human curation, templating, and translation. The project contributes four resources spanning multilingual annotation, datasets, evaluation, and participatory research.
- Motivation: Instruction fine-tuning datasets have focused almost entirely on English, despite thousands of languages being spoken worldwide.
- Motivation: Translation and automatic curation can introduce translation bias, cultural mismatch, noise, and limited instruction diversity.
- Aya Dataset: 2,997 collaborators across 119 countries produced 204,114 human-curated instruction-completion annotations in 65 languages.
- Aya Collection: 513M instances across 114 languages form the Aya Collection, created by templating 44 datasets and translating datasets into 101 languages.
- Contributions: The project releases four open resources: the Aya Annotation Platform, Aya Dataset, Aya Collection, and Aya Evaluation Suite.
- Aya Evaluation Suite: The Aya Evaluation Suite combines original and post-edited human prompts with translated prompts for multilingual open-ended generation.
2 Aya Annotation Platform & Aya Dataset
The Aya Annotation Platform supports decentralized multilingual data collection by contributors worldwide. The project documents contributor access, supported languages, device accessibility, and user demographics.
- Platform scope: The platform accommodates contributors across 119 countries and 134 languages, including dialects.
- Accessibility: Aya UI supports both desktop and mobile browsers, which accounted for approximately 54% and 46% of users respectively.
- Language support: Contributors select languages of proficiency during registration, while annotation support prioritizes 101 languages available in mT5.
- Contributors: The contributor pool includes 2,997 registered users across 134 languages and collects demographic information such as country, age, gender, and dialects.
- Demographics: More than two-thirds of users were aged 18–35; 68.1% identified as male, 28.5% as female, and 6.6% reported dialects.
- Geographic distribution: India had the highest number of registered users, with 346 of 2,997 users, while several countries had only one registered user.
Geographic-Based Contribution Assessment
Aya’s contributions were geographically uneven, with Asia and Africa providing most inputs. The platform combined original annotation, re-annotation, peer review, and filtering to assemble the released dataset.
- Regional distribution: 58.8% of contributions came from Asia and 27.4% from Africa, while Europe, Latin America, and other regions supplied the remaining 13.8%.
- Regional distribution: The project observed substantial regional skew and identified participation disparities as a topic requiring further research.The authors suggest opportunity cost, cultural beliefs about data sharing, and perceptions of technological service as possible factors.
- Annotation workflow: Contributors could write original prompt-completion pairs, edit existing examples, or provide feedback within a sequential find-fix-verify pipeline.The tasks were designed to collect new content, improve existing examples, and evaluate contribution quality.
- Contributor engagement: Annotators contributed for five days on average and 1.3 sessions, with activity generally declining after joining but a subgroup remaining active longer.
- Release criteria: Only re-annotations differing from originals by a character-level Levenshtein distance of at least 5 were released, and included languages required at least 50 contributions.These criteria aimed to avoid duplicate data while balancing quality and language coverage.
3 Analysis of Aya Dataset
The Aya Dataset contains 204,114 human-collected instruction-completion instances across 65 languages. Analysis shows that re-annotation increased completion length, longer examples correlated with higher approval, and contributions were unevenly distributed across languages and annotators.
- 204,114 instances cover 65 languages, including 22 high-resource, 12 mid-resource, and 31 low-resource languages.
- Length of Aya Dataset: 25% longer completions followed editing across data sources, while Aya original annotations became 40% longer on average.
- Length of Aya Dataset: Longer prompts and completions showed a positive correlation with average approval ratio, although length alone did not necessarily imply higher quality.
- Length of Aya Dataset: Completions varied sharply by language: they were 31% shorter than prompts in Japanese but 1258% and 2516% longer in Urdu and Yoruba.
- Length of Aya Dataset: Aya Dataset completions were considerably longer than those in other collections, despite being human-curated.
- Annotator Skew: Contribution distributions were uneven: Sindhi had one active annotator, while English and Portuguese had more than 80.
- The impact of introducing the Aya Score: After introducing the Aya Score, submitted completion lengths increased markedly as annotators were encouraged to expand short answers into full sentences or paragraphs.
4 Aya Collection
The Aya Collection combines templated, translated, and human-annotated instruction data. Its construction uses fluent-speaker templates, selective translation, permissive licensing, and quality-oriented dataset selection.
- Aya Collection: The collection targets multilingual model training and includes datasets selected for permissive licensing and redistribution.
- Templating Existing Datasets: Templated data uses fluent speakers to create instructions that expand existing datasets into multiple languages.
- Automatic Translation: Translated data comprises 19 hand-selected datasets translated into 101 languages and 114 dialects with NLLB.
- Aya Collection: The Aya Dataset is the collection’s only dataset that is entirely human-annotated.
- Aya Collection: Dataset selection prioritized diverse tasks, natural complete sentences, and exclusion of likely single-word answers or noisy automatically created data.
- Templating Existing Datasets: Aya is described as the first broad effort to involve fluent speakers in creating language-specific prompts for instruction-tuning expansion.
- Automatic Translation: Translation quality averaged 48.17 ChrF++ on FLORES, and errors led to excluding Standard Arabic in Latin script and Minangkabau in Arabic script.
5 Analysis of Aya Collection
The Aya Collection contains 513 million instances across 114 languages, with broad task coverage and attention to language balance, length, and perceived quality. Its analyses find substantial variation across languages and higher approval for Aya original annotations.
- Aya Collection: 513M instances span 114 languages across 44 templated and 19 translated datasets.
- Tasks Covered Across Templated and Translated Datasets: The collection covers Question Answering, Natural Language Generation, and Text Classification, with finer-grained task types within NLG and classification.
- Language Balance: The templated subset broadens representation for languages with limited digital data, although instance counts still vary across languages.
- Prompt and Completion Lengths: Low-resource languages occupy both ends of the length distribution, with Somali and Amharic showing longer average completions than some medium- and high-resource languages.
- Quality Assessment of All Different Data Sources: Aya original annotations had the highest average approval ratio at approximately 0.81, compared with approximately 0.50 for xP3.
6 Aya Evaluation Suite
The Aya Evaluation Suite addresses the gap in multilingual evaluation for open-ended LLM generation by combining human-curated, translated, and human-edited examples. It balances broad language coverage against the quality constraints of translated evaluation data.
- The suite targets multilingual open-ended generation, including brainstorming, planning, and other unstructured long-form responses.
- It combines human-curated examples in limited languages with automatic translations and human-post-edited translations in fewer languages.
- 1,750 instances across 7 languages form the aya-human-annotated test set, sampled from original annotations for balanced language representation.
- 200 culturally neutral Dolly prompts form an additional translated evaluation set after reviewers filtered an initial sample of 500 English prompts.
- Automatically translated evaluations cover 101 languages and 114 dialects, but their expressiveness depends on translation quality and may distort ability estimates.
7 A Participatory Approach to Research
Aya uses participatory, open-science collaboration to involve language experts and contributors worldwide in dataset creation. The approach broadens linguistic representation while making the dataset’s provenance, social context, and practical constraints visible.
- Aya’s participatory research model responded to narrow collaboration networks and the undervaluation of dataset creators’ contributions.
- The project used broad cross-institutional collaboration to collect linguistic data and engage communities in developing language technologies.
- Asynchronous communication and open meetings supported collaboration among researchers across time zones during the year-long project.
- Open access enabled participation by language experts without requiring formal machine-learning credentials, producing more representative multilingual data.
- Contributors’ motivations centered on community, identity, and social justice rather than financial remuneration.
- Contributor interactions shaped dataset provenance through debates over language representation, dialect distinctions, and intergenerational language knowledge.
- Power outages, unstable internet, armed conflict, and shipping barriers constrained participation and prevented gifts from reaching some contributors.
8 Related Work
Prior multilingual resources span unstructured corpora, translation datasets, task-specific benchmarks, instruction-tuning collections, and participatory projects. Aya builds on these lines of work while emphasizing multilingual instruction data and broad participatory curation.
- Multilingual corpora support unsupervised pre-training or machine translation, while task-specific datasets target capabilities such as cross-lingual understanding.
- Large multilingual corpora can suffer from inadequate quality and require extensive cleaning, whereas smaller task-specific datasets offer targeted, higher-quality data.
- NLLB released translation resources for more than 200 languages, while relatively fewer efforts focus on creating or curating low-resource-language data.
- Instruction-tuning datasets include human-curated, templatized, and language-model-generated instruction-response pairs.
- Prior multilingual instruction-tuning work largely relied on translating English datasets, templates, or human curation outside English.
- Template-based approaches can limit task diversity when the same template is reused across languages, motivating human-created non-English instructions.
- Machine translation expands language coverage but may lack variability and cultural nuance, while participatory projects have often targeted specific regions or tasks.
- Open-science initiatives such as BigScience demonstrate related efforts to address limitations in language-model development through inclusive collaboration.
9 Limitations of our work
Aya expands multilingual instruction-tuning resources but remains limited by incomplete language and dialect coverage, uneven contributions, cultural and linguistic variation, and possible unsafe or mislabeled data.
- Language and dialect coverage: 93% of the world’s languages are still not being used to train LLMs, and many dialects are absent from Aya.The dataset covers 65 languages and the collection 114, but only a small fraction of global linguistic diversity is represented.
- Language and dialect coverage: Aya excludes programming languages and keeps languages isolated, limiting representation of code-switching and regional variation.The project notes that multilingual speakers often code-switch between languages or dialects, but this was not represented in the dataset.
- Uneven distribution of contributions: Contributions are uneven across annotators and languages, with some languages dominated by one or two frequent contributors.A long tail of annotators supplied only one or two annotations, while relatively few contributors accounted for most annotations.
- Cultural or personal bias: Some language data may reflect narrow cultural viewpoints or content distributions shaped by the availability of online sources.The paper gives French and African-language news data as examples of potentially incomplete cultural or topical representation.
- Linguistic variation: Gendered pronouns, formality distinctions, and differing standards require context-sensitive choices that vary across languages and regions.The paper highlights gender-neutral reference, honorifics, and differences such as European versus Brazilian Portuguese.
- Data quality and safety: Human review reduced but did not eliminate offensive content, while missing relabeling tools could leave annotations under incorrect language categories.The platform had no dedicated toxicity flags and no mechanism for re-labeling assigned languages.
10 Conclusion
Aya combines participatory multilingual data creation with large-scale transformed and translated resources. The project releases open resources intended to support multilingual NLP research and future participatory collaboration.
- 10 Conclusion: The project treats participatory research as a basis for collaboration between multilingual contributors and NLP researchers.The authors describe continued opportunities for collaboration with sociolinguists, anthropologists, sociologists, and related researchers.
- 10 Conclusion: 2,997 collaborators from 119 countries helped build Aya’s human-curated dataset over eight months.The dataset contains 204,114 prompt-completion pairs covering 65 languages.
- 10 Conclusion: The Aya Collection contains 513M prompt and completion pairs covering 114 languages.It comprises 44 instruction-style datasets transformed from existing NLP datasets and expanded through translation into 101 languages.
A.1 Engagement Strategies
Aya used points, recognition, leaderboards, challenges, and progress updates to encourage sustained annotator participation. Registration patterns also varied across countries and changed after Google SSO was introduced.
- Engagement Strategies: Points, social-media recognition, leaderboards, mini-challenges, and sprints formed Aya’s engagement strategy.These mechanisms were intended to motivate contributors and foster friendly competition and community.
- Engagement Strategies: 500, 1,000, and 5,000 contributions marked reward milestones for certificates and Aya project apparel.Packages varied according to contribution level, with the most active contributors also publicly acknowledged.
- Engagement Strategies: The Aya Discord Bot tagged the daily top 10 contributors and aggregated regional contribution totals.These messages showed annotators how the dataset was growing across languages.
- Engagement Strategies: Daily, weekly, language-specific, and overall leaderboards ranked annotators according to their Aya score.The leaderboard provided multiple views of contributor performance.
- Engagement Strategies: After Google SSO was introduced, the number of registered users from several new countries increased significantly.Registration access differed by country, and Figure 20 compares Discord and Gmail sign-ups in the top 10 countries.
B UI Tasks
Aya’s UI supported annotation, peer review, re-annotation, contributor scoring, regional coordination, and language-specific collaboration. These tasks and structures were designed to improve data quality and support contributors across many settings.
- B UI Tasks: Reviewers rated re-annotated prompts and completions from 1 to 5 against the original annotation.Ratings ranged from “Much worse than the original annotation” to “Much better than the original annotation.”
- B UI Tasks: Ratings below 5 allowed reviewers to modify prompt-completion pairs and explain their decisions through additional feedback.This provided both a correction mechanism and a rationale for the selected rating.
- B UI Tasks: The Aya Score ranked annotators using peer-review-related contribution measures and weighted components.The score was introduced to favor high-quality submissions and encourage competitive participation.
- B UI Tasks: E counts edited re-annotations, C counts original annotations, and Q̂ is the average peer-review quality rating.Additional terms capture original annotations receiving positive feedback and feedback-covered annotations.
- B UI Tasks: Annotation criteria required clear prompts, correct and coherent completions, clean language, and non-generated original annotations.Re-annotation was used when prompts or completions needed editing, with particular encouragement to add detail and context.
- B UI Tasks: Regional and language-specific meetings coordinated contributors, onboarding, troubleshooting, datathons, and technical updates.Regional Leads and Language Ambassadors supported collaboration through recurring meetings and communication channels.
E Language Groupings
Aya groups languages into lower-, mid-, and higher-resourced categories using recorded, written, and catalogued NLP resources as a proxy for data availability. The grouping supports aggregate analysis but is explicitly imperfect as a universal classification.
- Language Groupings: Languages are grouped as lower-, mid-, or higher-resourced according to their recorded, written, and catalogued NLP resources.The categories aggregate five resource clusters from LDC, ELRA Map, and Wikipedia data.
- Language Groupings: The resource grouping is used as a proxy for data availability in pretraining and instruction-finetuning.
- Language Groupings: The classification is imperfect because languages and varieties cannot be universally classified along a single resource dimension.
- Annotation Process: Annotators were recruited based on self-reported native or professional proficiency in the languages required for the project.
- Annotation Process: Each translation was post-edited by one annotator and assessed by a second annotator, with 3–4 annotators involved in each task.
F.4 Post-Editing Effort
The post-editing analysis evaluates machine-translated prompts using edit-based metrics and finds substantial translation problems across all six languages examined. Russian required the most editing, while French required much less despite a similar share of prompts being edited.
- Post-Editing Effort: At least 41% of prompts were edited in every language, indicating that translation errors were common in the six-language test set.The analysis uses the overall edited-prompt percentage together with HTER and HChrF.
- Post-Editing Effort: Russian had the largest post-editing effort, with an average HTER of 37.43.HTER measures the percentage of words edited from the original machine translation.
- Post-Editing Effort: French had 5.56 HTER despite a similar edited-prompt ratio to Russian: 84.5% versus 86.5%.
- Translation Procedure: NLLB translations were sentence-tokenized and concatenated because the model could cut off abruptly when translating full paragraphs.Samples containing NLLB-introduced <unk> tokens were also filtered.
- Translation Quality: Figure 22 reports NLLB translation quality with ChrF++ scores on FLORES for English-to-Aya-language translations, grouped by resourcefulness.
J Data Cards
The data cards document three open multilingual resources: the human-curated Aya Dataset, the larger templated and translated Aya Collection, and the Aya Evaluation Suite. Together, they specify their construction, coverage, contributors, maintenance, and prompt formats.
- Aya Dataset: The Aya Dataset contains 204,114 annotated prompt-completion pairs curated by an open-science community.
- Aya Dataset: The Aya Dataset covers 65 languages and was created through volunteer annotations on the Aya Annotation Platform.
- Aya Collection: The Aya Collection applies fluent-speaker templates to 44 datasets and translates 19 instruction-style datasets into 101 languages.
- Aya Collection: The Aya Collection provides 513,579,625 prompt-completion instances covering a wide range of tasks.
- Release and Maintenance: The resources are open-sourced under the Apache 2.0 license and made available for download.
- Aya Evaluation Suite: The Aya Evaluation Suite contains 25,750 open-ended conversation-style prompts covering 101 languages across human-written, machine-translated, and human-edited subsets.
- Aya Evaluation Suite: The evaluation suite combines original volunteer annotations, automatic translation, and professional post-editing to balance language coverage with human-sourced quality.