PaperlayerView exact revision on arXiv

Source-linked AI summary

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, Sara Hooker

arXiv:2402.07827v1cs.CL

TL;DR

Aya addresses multilingual language-model development centered on data-rich languages by open-sourcing an instruction-finetuned model covering 101 languages and evaluating it across 99. It outperforms mT0 and BLOOMZ on most tasks, including 13.1% and 11.7% relative gains over mT0x2 on discriminative and generative tasks.

  • Problem

    Multilingual language-model datasets and progress have coalesced around a few data-rich languages, leaving broader linguistic representation underdeveloped.

    Source: §1 · p. 1

  • Method

    Aya combines multilingual instruction finetuning with pruned, provenance-focused data and human annotations to build an open-source model covering 101 languages.

    Source: §1 · p. 3, §2 · p. 5, §2.1 · p. 6

  • Results

    13.1% and 11.7% relative gains over mT0x2 were reported on discriminative and generative tasks, while Aya improved across automatic and human evaluations.

    Source: §1 · p. 3, §11 · p. 42

  • Takeaways & Limitations

    Aya broadens multilingual instruction finetuning and evaluation to languages and safety dimensions that have received less representation in open-source model development.

    Source: §11 · p. 42

    Aya’s 13-billion-parameter size limits its usability on many consumer-grade hardware.

    Source: §9 · p. 40

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101

1 Introduction

Aya addresses the concentration of NLP progress in data-rich languages by instruction-finetuning an open-source model across 101 languages, including 51 lower-resourced languages. It expands multilingual training and evaluation while investigating data balancing and multilingual safety.

Motivation

  • Aya targets linguistic inequality caused by NLP datasets and instruction-tuning resources that disproportionately favor data-rich and English languages [∀et al., 2020a; Bird, 2022; Longpre et al., 2023b].This imbalance contributes to performance disparities, bias, safety flaws, and higher token and latency costs for marginalized languages [Kunchukuttan et al., 2021; Schwartz et al., 2022; Kotek et al., 2023; Khandelwal et al., 2023; Vashishtha et al., 2023; Khondaker et al., 2023; Yong et al., 2023a; Nasr et al., 2023; Li et al., 2023c; Lukas et al., 2023; Deng et al., 2023; Ji et al., 2023b; Cui et al., 2023; Ahia et al., 2023].Source: §1 · pp. 1–3

Approach

  • Aya addresses the fundamental obstacle of securing comprehensive multilingual IFT datasets by fine-tuning a 13B-parameter mT5 model on an instruction mixture including 101 languages [Xue et al., 2020].Source: Figure 1 · p. 2, §1 · p. 3

Expansion of Language Coverage

  • Aya expands instruction-finetuning coverage to 101 languages, including 51 lower-resourced languages, compared with 46 for xP3, mT0, and BLOOMZ.Its 203M-example training mix is 2.5× xP3’s 81M examples and reduces English representation from 39% to 21.5%.Source: §1 · p. 3

Broadening Multilingual Evaluation

  • Aya achieves 13.1% and 11.7% relative gains over mT0x in discriminative and generative tasks, respectively, and a 75% human preference win rate across 7 languages.Evaluation covers 99 languages through discriminative, generative, simulated win-rate, human, and safety benchmarks.Source: §1 · p. 3

Data Weighting and Pruning

  • Aya rebalances training through pruning and source-weighting ablations, removing 19.66% of English and 18.25% of multilingual instances while varying translated, templated, and human-annotated data.The final mixture includes filtering, language filtering, and subsampling across its data collections.Source: §1 · pp. 3–4, Table 1 · p. 4

Safety

  • Multilingual safety context distillation reduces harmful generations for adversarial prompts by 78–89%, while Aya’s toxicity and social- and gender-bias risks are analyzed across 18 languages.Aya is released as an open-source model under the Apache 2.0 License.Source: §1 · p. 4

2 Data

Aya addresses multilingual instruction-tuning data scarcity and quality through curated, provenance-conscious mixtures combining templates, human annotations, translation, and synthetic generation. The resulting resources span 101 languages while explicitly distinguishing higher-, mid-, and lower-resource coverage.

2 Data

  • Aya’s data strategy combines aggregation, pruning, human-curated templates and annotations, machine translation, and synthetic data generation to improve multilingual instruction-data availability.The selection emphasizes high quality, prompt-type and task diversity, and provenance compatible with an Apache 2.0 release.Source: §2 · pp. 4–5

2.1 Multilingual Templates

  • 680M xP3x examples across 277 languages and 16 tasks are pruned to 168M examples across 101 languages and 56 datasets, with human review removing low-quality templates.The final pruning removes 50.2% of English and 35.9% of multilingual templates, while average instruction lengths increase 7.0% and 16.8%, respectively.Source: §2.1 · p. 6, Pruning xP3x · p. 7

2.2 Human Annotations

  • The Aya dataset contributes 204K native-speaker prompt-response pairs collected through a year-long initiative involving 2,997 participants from 110 countries.After filtering, 199.5K samples remain across 64 training languages, including 22 higher-resource, 12 mid-resource, and 30 lower-resource languages.Source: §2.2 · p. 8

2.3 Augmentation via Automatic Translation

  • Aya’s translated mixture contains 7.5M instances across 93 languages, using per-language, per-dataset sampling of up to 3,000 examples to preserve instance-level diversity.Translation expands coverage but can introduce translationese, whose performance trade-off remains insufficiently understood and is examined later in Section 5.6.Source: §2.3 · pp. 8–9

2.4 Synthetic Data Generation

  • ShareGPT-Command provides 6.8M synthetically generated and machine-translated examples across 93 languages, combining human-annotated prompts with synthetic English completions from Command.Prompts are filtered for URLs, excessive length, and non-English content before translation and low-quality pruning.Source: §2.4 · p. 9

3 Experimental Set-up

The experiments finetune mT5 across 101 languages, investigate source- and dataset-level sampling, and compare Aya with multilingual models spanning architectures, sizes, languages, and training regimes. Evaluation distinguishes unseen discriminative tasks and multilingual MMLU from seen generative tasks with held-out splits.

3.1 Pre-trained Models & Finetuning

  • Aya finetunes the 13B-parameter mT5 model, pretrained on 1 trillion tokens across 101 languages, primarily to preserve broad multilingual coverage during instruction finetuning.The authors note that mT5 is older and less powerful than newer models, while the Aya dataset release provides 513M multilingual instruction-tuning instances [Singh et al., 2024].Source: §3.1 · p. 10

3.2 Data Sampling Ablations

  • Sampling ablations address a 25M-instance budget and severe source imbalance by varying source weights and optional dataset-level weights across human-annotated, translated, and templated data.Human annotations comprise 0.7% of the combined 203M instances, and the human-annotation-heavy variant upweights the Aya Dataset to at most 25%.Source: §3.2 · p. 11

3.3 Baselines

  • Baselines span 13B–176B models, Llama, BLOOM, and mT5 bases, 26–101 languages, and supervised or preference-training regimes, including mT0, BLOOMZ, mT0x, Bactrian-X, Okapi, and Aya Safe.mT0 and BLOOMZ cover 46 languages, whereas mT0x extends xP3 to 101 languages for a fairer same-base comparison; Aya Safe is evaluated for safety-performance trade-offs.Source: §3.3 · p. 12
  • Evaluation separates unseen tasks, including four discriminative tasks and multilingual MMLU, from seen generative tasks evaluated on held-out validation and test splits.This design distinguishes performance on tasks entirely excluded from training from performance on supervised task families with held-out instances.Source: Table 4 · p. 13

4 Evaluation

Aya’s evaluation suite broadens multilingual assessment across unseen, in-distribution, generative, human-preference, and simulated-preference settings, covering 99 of 101 training languages. It combines benchmark diversity with professional annotation and GPT-4 judging to assess both task performance and open-ended generation.

4 Evaluation

  • The evaluation suite covers zero-shot discriminative tasks, 5-shot Multilingual MMLU, in-distribution datasets, human preferences, and LLM-simulated win rates across varied languages.Table 4 summarizes the evaluation tasks, datasets, and language coverage.Source: §4 · p. 13
  • Evaluation covers 99 of Aya’s 101 training languages, excluding Frisian and Latin, compared with 27 languages in prior massively multilingual work.Most tasks still cover only 10–15, often higher- or mid-resource languages; FLORES-200 and XLSum provide the broadest coverage.Source: §4 · p. 13

4.1 Discriminative Tasks

  • Unseen discriminative evaluation uses XWinograd, XNLI, XCOPA, and XStoryCloze across coreference resolution, sentence completion, and natural language inference, enabling comparison with mT0 and BLOOMZ.The tasks are held out from training, while Multilingual MMLU provides 5-shot general language-understanding evaluation across 31 translated languages.Source: §4.1 · p. 14

4.2 Generative Tasks

  • Generative evaluation uses FLORES-200, XLSum, and TydiQA GoldP for translation, summarization, and question answering, with FLORES-200 and XLSum covering 99 languages.Aya is compared with mT0x on devtest or validation splits because other baselines include evaluation data or lack the evaluated language coverage.Source: §4.2 · p. 14

4.3.1 Human Evaluation Protocol

  • Human evaluation uses compensated professional annotators in seven languages to choose between model completions, quantify qualitative differences, validate simulated preferences, and identify generation errors.Pairs are rated once with ties allowed but discouraged, and a subset is double-annotated to estimate label variance and calibrate LLM-judge agreement.Source: §4.3.1 · p. 15

4.3.2 Simulated Preferences

  • Simulated preferences use GPT-4 to measure pairwise win rates between Aya and mT0 or mT0x across 10 mixed-resource languages, with Bactrian-X comparisons on five languages.The evaluation uses held-out 200-prompt Dolly translations, defaults to human-edited prompts where available, and uses 250 human-annotated prompts per language for Bactrian-X.Source: §4.3.2 · p. 15

5 Results

Aya generally outperforms multilingual baselines across unseen discriminative, MMLU, generative, preference, and human evaluations, while results show that data-mixture composition and model scale strongly affect capabilities. Performance advantages vary by task and language-resource group, and open-ended generations still exhibit errors and hallucinations.

5.1.1 Unseen tasks

  • 75.12%: Aya template-heavy outperforms mT0, BLOOMZ, and Bactrian-X by an average of 19.8% across unseen discriminative tasks despite covering 101 languages.Aya’s best variant also outperforms the equal-coverage mT0x by 14.8% over 101 languages, indicating that data composition rather than capacity dilution explains much of the gap.Source: Table 5 · p. 16, §5.1.1 · pp. 16–17

5.1.2 Multilingual MMLU

  • 21.1%: Aya improves average multilingual MMLU accuracy over mT0x, 18.4% over mT0, and 25.1% over Okapi, achieving the best overall performance across 26 languages.Aya, mT0x, and mT0 use 5-shot evaluation, whereas Okapi uses 25-shot evaluation.Source: Table 6 · p. 17, §5.1.2 · p. 17

5.2 Generative Tasks

  • 44% and 31%: Aya translation-heavy achieves the highest improvements over mT0x on FLORES-200 for X →English and English →X, while gains are 1.8% RougeLsum on XLSum and 2.2% F1 on Tydi-QA.Template-heavy instead performs best on the narrower XLSum and Tydi-QA evaluations, with 7.4% RougeLsum and 3.5% F1 improvements, respectively.Source: Table 7 · p. 18, §5.2 · p. 18

5.3 Performance Comparison by Language Resourcedness

  • 40.8%: Aya improves FLORES-200 over mT0x by 7.8 spBLEU points on average, with relative gains of 36.1% for HR, 34.9% for MR, and 47.1% for LR languages.Aya also improves unseen discriminative tasks and multilingual MMLU in every resource group, with the largest differences in HR languages at 12.1% and 21.8%, respectively.Source: Figure 3 · p. 19, §5.3 · p. 19

5.4 Simulated Win Rates and Human Eval

  • 87% and 86%: GPT-4 prefers Aya over mT0 and mT0x across all evaluated languages, while human raters prefer Aya 77% of the time on average.Against Bactrian-X, Aya wins 82% of the time outside English, although evaluators report repetitions, hallucinations, incoherence, grammar and factuality errors, especially in some languages.Source: §5.4 · pp. 19–20, Figure 4 · p. 20, Figure 6 · p. 20

5.5 Tension between Discriminative Tasks and Open Ended Generations

  • 89% and 80%: Aya is preferred over mT0 in simulated and human evaluations, showing that strong discriminative performance does not directly predict open-ended generation quality.mT0 produces much shorter completions than Aya, averaging 49 versus 310 characters, and GPT-4 ratings may be affected by verbosity and salience bias.Source: §5.4 · p. 21, §5.5 · p. 22, Figure 7 · p. 22

5.6.1 The Impact of Sampling Weights

  • 47% versus 31%: translated-heavy beats template-heavy in simulated open-ended win rates, whereas template-heavy performs best on discriminative tasks and multilingual MMLU; translated-heavy is therefore selected as final Aya.Human-annotated upsampling commonly performs worst, plausibly because its 199.5K instances are repeated more than 30 times at a 25% sampling weight, inviting overfitting.Source: Figure 8 · p. 23, §5.6.1 · pp. 23–24

5.7 Contribution of Individual Data Sources / 5.7.1 Model size matters

  • 73.9%: increasing model size from 1.2B to 13B raises unseen-task accuracy from 45.9% to 73.9%, while adding multilingual templates mainly improves discriminative tasks, translated data mainly improves FLORES, and every source improves open-ended generation.The consistent scaling gains suggest that the 13B model remains under-capacity for the multilingual setting.Source: §5.7 · p. 24, Figure 10 · p. 24, §5.7.1 · pp. 24–25

6 Safety Mitigation

Aya is vulnerable to harmful adversarial prompts across languages, while multilingual safety context distillation sharply reduces harmfulness. The mitigation incurs modest general-task losses and leaves refusal quality and multilingual safety coverage as open limitations.

6 Safety Mitigation

  • Preamble-based safety evaluations are necessary across languages because English-only mitigation can leave safety leaks, but the study covers adversarial harmful intent rather than professional-level jailbreaks.The authors note that broader multilingual safety benchmarks and metrics remain needed, and release Aya to enable community red-teaming.Source: §6 · p. 25

6.1.1 Mitigation With Preambles

  • A translated safety preamble makes Aya reject 88% of multilingual AdvBench harmful requests on average, but increases toxic open-ended completions by around 19%, so it should be restricted to harmful contexts.Refusal rates are lowest for Scottish Gaelic at 72% and Hindi at 77%.Source: §6.1.1 · p. 26

6.1.2 Safety Context Distillation with Synthetic Refusals

  • Safety context distillation extends preamble-based safety mitigation to multilingual Aya by finetuning the model on teacher-generated refusal prompts across languages.The authors describe this as the first multilingual extension of safety context distillation; synthetic refusals are generated from harmful prompts with translated language coverage and paired with prompts without preambles.Source: §6.1.2 · p. 26

6.2 Experimental Setup

  • Without mitigation, Aya produces harmful outputs for 89–90% of human-evaluated adversarial prompts across Arabic, English, and Hindi, with evaluation using held-out multilingual AdvBench prompts.The setup trains Aya Safe for 30k steps using distilled safety data weighted at 3% and evaluates 120 held-out prompts per language; GPT-4 ratings agree 93% with human ratings on average.Source: §6.2 · p. 27, §6.3 · p. 28

6.3 Safety Mitigation Results

  • Aya Safe reduces harmful outputs across languages to 4–11% by human ratings and 1–10% by GPT-4, versus 5% average harmfulness after mitigation compared with 12% for the teacher.Hindi retains the highest harmfulness: 11% by human ratings and 13% by GPT-4.Source: Figure 12 · p. 28, §6.3 · p. 28

6.4 Trade-offs between Performance and Safety

  • Safety context distillation causes 0.2–3.2-point losses on general benchmark evaluations, while toxicity and bias performance is comparable or marginally improved; Aya Safe’s helpfulness is comparable to Aya despite GPT-4 preferences of 50% versus 38%.Source: §6.4 · pp. 29–30

7 Benchmarking Toxicity and Bias

Aya and Aya Safe were evaluated against mT0x for toxicity and bias across 18 languages and five language families, using multilingual open-ended generation and gender-bias translation tests. Aya models generally showed lower toxicity, while Aya Safe offered stronger translation accuracy and reduced gender disparities despite persistent male and pro-stereotypical tendencies.

7 Benchmarking Toxicity and Bias

  • Across 18 languages and five language families, the analysis compared Aya and Aya Safe with mT0x on open-ended toxicity, identity-group toxicity, and gender bias in translation.The toxicity tests covered 14 languages in multilingual RealToxicityPrompts-style evaluations, while translation tests used Wino-MT across eight target languages.Source: §7 · p. 30

7.1.1 Non-Toxic Open-Ended Prompts

  • Aya and Aya Safe had lower expected maximum toxicity and toxicity probability than mT0x across all 14 evaluated languages; each averaged 8% toxicity probability over k = 25 generations.Their average worst-case toxicity levels were 22% for Aya and 24% for Aya Safe.Source: Figure 14 · p. 31, §7.1.1 · p. 32

7.1 Toxicity and Bias in Open-Ended Generation

  • Toxicity was measured with Expected Maximum Toxicity and Toxicity Probability, both lower-is-better metrics based on Perspective API scores above 0.5.The open-ended evaluations sampled k = 25 outputs for non-toxic prompts and k = 800 outputs for identity-group prompts.Source: §7.1 · p. 31, §7.1.1 · p. 32, §7.1.2 · p. 32

7.1.2 Toxicity Propensity Given Different Population Groups

  • Aya and Aya Safe generally had lower toxicity than mT0x for identity-group prompts, except in English, where Aya was more toxic for several racial identity groups.Aya Safe also had lower toxicity probability than mT0x, while Aya exceeded mT0x for Asian people, White men, and Indian men in English.Source: §7.1.2 · pp. 32–33, Figure 15 · p. 33

7.2 Gender Bias in Machine Translation

  • Aya Safe achieved the highest overall translation accuracy for Romance languages, while mT0x led for Semitic languages; Aya models had similar accuracy ranges overall.The comparison covered Spanish, French, Italian, Russian, Ukrainian, Hebrew, Arabic, and German using Wino-MT.Source: §7.2 · pp. 33–34, Table 9 · p. 35
  • Aya and Aya Safe remained prone to male-gender and pro-stereotypical translations, but Aya Safe showed the strongest overall signs of reduced gender disparity; mT0x had the lowest average stereotype gap.Aya Safe had the lowest ΔG in Romance languages and German, while mT0x had the lowest average ΔS by a small margin.Source: §7.2 · p. 35, Table 10 · p. 35

8 Related Work

This section situates Aya within multilingual data initiatives, massively multilingual pre-training, instruction tuning, translation augmentation, dataset selection, and multilingual safety research. It emphasizes Aya’s focus on instruction following, broad target-language usability, human-informed translation augmentation, and safety across languages.

Language Diversity in Open-source Multilingual NLP

  • Open multilingual NLP initiatives expand datasets, benchmarks, tools, and models for low-resource translation and language-specific research, but coverage remains far below global linguistic diversity [Tiedemann, 2020].Examples include Masakhane [∀et al., 2020b], NusaCrowd [Cahyawijaya et al., 2022], Turkic Interlingua [Mirzakhalov, 2021], IndicCorp, IndicXtream, Masader, and SEACrowd24.Source: §8 · p. 36

Pre-trained Multilingual Models

  • Massively multilingual pre-trained models cover around 100 languages, but pre-training-centric systems provide limited off-the-shelf usability; Aya instead targets instruction following [Chung et al., 2023a].Adding languages one at a time through continued pre-training does not scale well, while concurrent work extends LLaMA 2 with vocabulary expansion and continued pre-training [Yong et al., 2023b; Luukkonen et al., 2023; Lin et al., 2024].Source: §8 · p. 37

Instruction Tuning

  • Multilingual instruction-tuning research shows that including target languages improves performance, yet BLOOMZ and mT0 remain limited beyond their 46 finetuning languages [Muennighoff et al., 2023d].Aya addresses broader coverage, diverse target-language tasks, open-ended generation, and prompts written in the task language rather than code-switched English prompts [Huang et al., 2023a].Source: §8 · p. 37

Translation Augmentation

  • Translation augmentation broadens multilingual training data, while Aya combines machine translation with community quality filtering and human translations to add target-language coverage and cultural nuance.Machine-translated prompts can lack variability and cultural nuance, but remain useful for languages with limited training data [Urbizu et al., 2023; Lin et al., 2021].Source: §8 · pp. 37–38

Dataset Weighting

  • Prior work dynamically weights, selects, or prunes pre-training and finetuning data using domain objectives, language scores, perplexity, error norms, and model-discriminating criteria [Xie et al., 2023; Wang et al., 2020b].These methods aim to improve efficiency, performance, or annotation effort across multilingual and general training settings.Source: §8 · p. 38

Evaluation of Toxicity and Bias in LLMs

  • Existing toxicity and bias evaluations are predominantly English or cover few languages, while multilingual safety studies find lower-resource prompts can bypass safeguards; Aya reports safety improvements from context distillation without alignment training [Deng et al., 2023; Shen et al., 2024].Prior multilingual toxicity work often evaluates detection rather than generation, and multilingual training mixtures can worsen the harmlessness-helpfulness tradeoff [Anil et al., 2023].Source: §8 · pp. 38–39

9 Discussion

Aya’s discussion emphasizes that multilingual scale requires substantial capacity but remains constrained by model choice, incomplete language and dialect coverage, cultural and linguistic biases, and safety-evaluation limitations. The authors identify compression, broader data coverage, culturally diverse safety measures, and improved evaluation as directions for further work.

Model Choice

  • Aya’s mT5 foundation enabled broad multilingual coverage, but its outdated knowledge, weaker performance, and 101-language limit constrain future use.The authors note that no available model then matched mT5’s performance while covering more languages.Source: §9 · pp. 39–40

Model Size

  • The 13B size mitigates multilingual capacity dilution, but limits consumer hardware usability and leaves performance–cost trade-offs unresolved.Quantization and pruning may reduce computational cost, but their multilingual performance effects require further research.Source: §9 · p. 40

Language and dialect coverage

  • Although Aya covers 101 languages, 93% of the world’s languages remain absent from LLM training, while many dialects and code-switching patterns are unrepresented.The model treats languages as isolated for classification and downstream language-specific applications, simplifying multilingual variation.Source: §9 · p. 40

Model values

  • Aya’s data may overrepresent Global North cultures, regional annotator perspectives, and news-style language rather than everyday usage.French data, for example, may emphasize France while underrepresenting Québec, Togo, or Senegal.Source: §9 · p. 40

Model behavior

  • Gendered pronouns, honorific systems, and translation differences can cause Aya to produce insufficiently neutral, standardized, or contextually formal responses.The authors specifically highlight languages including Arabic, Japanese, Yoruba, French, Spanish, and German as requiring context-sensitive choices.Source: §9 · pp. 40–41

Safety measures & mitigation

  • Safety context distillation improves refusal of malicious prompts, but has limited effects on open-ended toxicity and reflects an Anglo-centric view of harmfulness.Safety prompts translated from English may not capture culturally specific harms, limiting applications such as hate-speech prevention.Source: §9 · p. 41

Toxicity and bias analysis

  • Aya’s toxicity and bias analysis covers the most languages to date but remains limited by resource coverage, translated prompts, and Perspective API bias and drift.Lower-resourced-language gender bias is outside current coverage, while translated RealToxicityPrompts serve only as proxies for real-world interaction.Source: §9 · p. 41

10 A Participatory Approach to Research

Aya demonstrates that broad, cross-institutional global collaboration can enable multilingual language-model research beyond narrow networks. The initiative builds on open-science and inclusive collaboration efforts aimed at creating a lasting, diverse research ecosystem.

Motivation and contribution

  • Aya was made possible by broad cross-institutional, global collaboration, addressing the limitations of narrow research networks concentrated in a few institutions and regions [Nakamura et al., 2023; Park et al., 2023].The paper links small, specialized collaboration networks to hindered innovation and presents Aya as their collaborative alternative.Source: §10 · pp. 41–42

Open-science ecosystem

  • Aya follows open-science initiatives such as BigScience, whose global and inclusive collaboration produced BLOOM [Workshop et al., 2022] and the ROOTS corpus.The initiative is situated alongside later open-science efforts and wider collaborative communities spanning multiple regions and organizations.Source: §10 · p. 42

11 Conclusion … B.1 Pruning xP3x

This section summarizes Aya’s multilingual development, evaluations, open resources, and safety considerations, while documenting the covered languages and the review process used to prune low-quality or repetitive xP3x templates.

11 Conclusion

  • Aya more than doubles languages covered in instruction finetuning, evaluation, and safety, releasing its resources under permissive open-source-compliant licenses and improving over massively multilingual open-source models across automatic and human evaluations.The initiative involved over 3000 collaborators from 110 countries and characterized bias, toxicity, and harm across languages.Source: §11 · p. 42

A Languages in Aya Model

  • The Aya model covers 101 languages, with each language classified by script, language family, subgrouping, and higher-, mid-, or lower-resource status.Table 11 identifies these attributes and cites [Joshi et al., 2020] for the resource classification.Source: Table 11 · p. 78

B.1 Pruning xP3x

  • xP3x template pruning sampled three examples per task and dataset, translated multilingual samples into English, and penalized short targets, repetition, and grammatical, structural, or coherence errors while preserving task diversity.Reviewers used multiple samples to assess task quality when a single example was ambiguous; preference favored long instructions and reduced 1–2 word targets as much as possible.Source: §B.1 · p. 78
  • Four reviewers independently labelled examples yes or no, justified exclusions with comments, and participated in reviewer resolution.Source: §B.1 · p. 78
  • Reviewers resolved disagreements by discussing the comments attached to each review and reaching a final decision.Source: §B.1 · p. 78

B.2 List of xP3x Datasets … C Data Distribution per Language for Sampling Variants

The appendices document the xP3x datasets filtered for Aya’s language coverage, the English and multilingual datasets/templates retained after pruning, the translated-data collection, and language distributions under alternative sampling weights. Together, these inventories specify the data sources, templates, and sampling variants used in the multilingual training mixture.

B.2 List of xP3x Datasets

  • The listed datasets cover a broad task mixture, including QA, summarization, code synthesis, sentiment analysis, topic classification, generation, and bug prediction.Dataset entries provide task labels and licensing information where available, including QA resources such as ARC-Easy, TyDiQA-GoldP, HotpotQA, and OpenBookQA.Source: §B.2 · pp. 79–80
  • The xP3x catalog spans both single-language and multilingual resources, including English, Chinese, programming, and datasets covering up to 200 languages.Examples include Flores with 200 languages, GEM/xlsum with 45, MultiEURLEX with 23, MLQA with 7, and TyDiQA-GoldP with 11.Source: §B.2 · pp. 80–81
  • The xP3x inventory is filtered to datasets whose languages match those used in the Aya model.Table 12 lists the resulting datasets, languages, licenses, tasks, and prompt/completion length statistics.Source: Table 12 · p. 82

B.2.1 English Datasets and Templates Preserved Post-Pruning

  • English post-pruning coverage retains diverse datasets and many templates spanning question answering, classification, reasoning, summarization, paraphrase, and sentiment tasks.The preserved list includes datasets such as COS-E, HotpotQA, Social IQa, BoolQ, QuAIL, SQuAD, App Reviews, TyDiQA, ARC, QQP, MRPC, and SciQ.Source: §B.2.1 · pp. 83–85, Table 13 · p. 86

B.2.2 Multilingual Datasets and Templates Preserved Post-Pruning

  • Multilingual post-pruning coverage preserves datasets and templates for translation, summarization, question answering, paraphrase, and cross-lingual classification.The inventory includes GEM/BiSECT, WikiLingua, XLSum, Tatoeba, TyDiQA, MLQA, PAWS-X, XQuAD, XLM-WIC, and FLORES templates.Source: §B.2.2 · p. 87, Table 14 · p. 88

B.3 List of Translated Dataset

  • The translated-data inventory combines the ShareGPT Command dataset with a translated subset from the Aya Collection.Table 15 defines the scope of the listed translated datasets rather than reporting performance or sampling outcomes.Source: Table 15 · p. 88

C Data Distribution per Language for Sampling Variants

  • Figure 18 compares the percentage of examples assigned to each language under different weighting schemes.The figure describes sampling variants through their resulting per-language data distributions.Source: Figure 18 · p. 89

D Simulated Preference Evaluation … E.5.1 Agreement between Human and GPT-4 Pairwise Ratings

The evaluation combines GPT-4 simulated preferences with multilingual human pairwise and harmfulness ratings, using calibrated annotation procedures and explicit criteria. GPT-4–human agreement ranges from 38.9% to 86.5%, decreasing for harder comparisons between similar models, especially Aya Safe.

D Simulated Preference Evaluation

  • GPT-4 simulated preference evaluation used multilingual prompts based on human guidelines, a system preamble, and randomized model order to reduce bias.The template asks GPT-4 to assess language adherence, task fulfillment, factual and semantic comprehensibility, grammar, and fluency.Source: D Simulated Preference Evaluation · pp. 89–90

E Human Evaluation

  • Human evaluation covered pairwise preferences and harmfulness ratings, with annotators recruited for native or professional proficiency in the required languages.Participants came from diverse social and professional backgrounds, and the study described safeguards for exposure to harmful content.Source: E Human Evaluation · p. 90, §E.1 · pp. 90–91

E.1 Annotators

  • Annotators were paid 30 CAD per hour and were selected for language proficiency, with efforts to include people representative of the supported communities.The pool spanned diverse socioeconomic backgrounds, careers, education levels, and self-reported identities.Source: §E.1 · p. 91

E.2 Annotation Process

  • Annotation procedures used author briefings, ongoing Slack discussion, and randomized presentation order, while most examples received only one annotation.Each example was rated by one annotator, with 3–4 annotators involved in each task; limited re-annotations targeted selected model comparisons.Source: §E.2 · p. 91

E.3 Instructions for Pairwise Ratings

  • Pairwise instructions prioritized task fulfillment, followed by grammar, semantic coherence, and creativity, while allowing Both good, Both bad, and incomprehensible-prompt labels.Ties were reserved for cases where annotators could not distinguish the completions meaningfully.Source: §E.3 · p. 92

E.4 Instructions for Harmfulness Ratings

  • Harmfulness ratings treated on-topic responses that encouraged or enabled harmful goals as harmful, while refusals were harmless and severely unintelligible outputs were nonsensical.Annotators were instructed to inspect the entire completion because a refusal followed by harmful instructions remained harmful.Source: §E.4 · p. 92

E.5.1 Agreement between Human and GPT-4 Pairwise Ratings

  • 38.9%–86.5% agreement was observed between GPT-4 and human pairwise ratings, with lower agreement for Aya Safe and higher agreement for mT0 and mT0x comparisons.GPT-4 tends to choose a model more often, whereas humans more frequently assign ties, amplifying disagreement on difficult comparisons.Source: §E.5.1 · p. 93, Table 16 · p. 93

E.5.2 Agreement between Humans in Pairwise Ratings … H.1 Sampling Fewer Outputs

The appendix examines human-rating reliability, generation quality, detailed multilingual results, and toxicity evaluation across translated prompts and sampled outputs. It finds fair human agreement, language-dependent toxicity differences, and qualitative strengths and weaknesses in Aya generations.

E.5.2 Agreement between Humans in Pairwise Ratings

  • Average human agreement was Cohen’s κ = 0.38 with a 67.4% agreement rate, and humans generally agreed more with each other than with GPT-4.This pattern held except for the Aya-versus-mT0x task in French; human agreement was less affected by task difficulty or ambiguity than GPT-4 agreement.Source: §E.5.2 · pp. 93–94, Table 17 · p. 94

E.6 Generation Quality Discussion

  • Aya generations were generally more verbose and elaborate than mT0(x), but showed grammar errors, repetitions, weak reasoning, translation artifacts, and enumeration defects across several languages.Annotators sometimes preferred shorter mT0(x) outputs, while GPT-4 did not appear similarly irritated by the artifacts in the example.Source: §E.6 · p. 94

F Detailed Results for Section 5

  • Table 18 reports per-language results for Aya’s templated-heavy, translated-heavy, and human-annotated-heavy variants alongside mT0x across all evaluations.The detailed results cover every language included in the general evaluation suite.Source: F Detailed Results for Section 5 · p. 94, Table 18 · p. 106

G.1 Translation of RTP prompts

  • RTP prompts were translated into 14 Perspective API-supported languages, excluding non-whitespace-separated and right-to-left languages because the automatic translation heuristic was unsuitable.Translated prompts were toxicity-screened because English non-toxic prompts could become toxic after translation and resplitting.Source: §G.1 · p. 106

G.2 Toxicity of Multilingual RTP input prompts

  • German, Hindi, Korean, and Portuguese had 5% more toxic translated prompts than the other languages, while 0.3% of English prompts were also scored toxic.The cross-language differences may reflect typological changes such as SVO versus SOV word order, and English scores may reflect Perspective API changes over time.Source: §G.2 · pp. 106–107

H.1 Sampling Fewer Outputs

  • Sampling 25 rather than 800 outputs per prompt produced toxicity patterns similar to RTP results: Aya, Aya-Safe, and mT0x were highest for German and Portuguese and lowest for French.For German, output toxicity ranked mT0x highest, followed by Aya, then Aya-Safe.Source: §H.1 · p. 107

H.2 Co-occurrence Analysis

The co-occurrence analysis examines adjectives and adverbs in Aya’s identity-group generations to investigate higher English toxicity, while limiting counts to the first full sentence. Aya shows undesirable and stereotypical racial co-occurrences, although racial terms are not geographically disambiguated, complicating interpretation.

H.2 Co-occurrence Analysis

  • The analysis counts adjectives and adverbs only in each generation’s first full sentence, reducing noise from descriptive words unrelated to the identity group, and identifies them with language-specific spaCy models.Models were specified for German, English, French, Dutch, Portuguese, Swedish, and Chinese using spaCy 3.7.2.Source: §H.2 · pp. 107–108
  • Because racial identities such as “Asian” and “White” are not disambiguated by geographic origin, undesirable co-occurrences cannot be mapped to precise racial subgroups.This limitation follows Chowdhery et al. [2022] and affects interpretation of the observed model generations.Source: §H.2 · p. 109

I Toxicity Detection Task … J.2 Harmful Prompts Data Collection

Aya leads toxicity detection across multilingual settings, while safety-mitigated finetuning reduces this capability. The safety-mitigation pipeline uses a one-step rejection preamble, carefully selected multilingual harmful prompts, and automatic filtering.

I Toxicity Detection Task

  • Toxicity detection is a zero-shot binary classification task evaluated with AUC-ROC from normalized log-likelihoods of “yes” versus “no” continuations.The multilingual Jigsaw set covers French, Italian, Portuguese, Russian, Spanish, and Turkish; English evaluation uses 10k CivilComments test comments.Source: I Toxicity Detection Task · p. 111
  • Aya is the best-performing toxicity detector, outperforming instruction-tuned and pretrained baselines and even PaLMv2 through cross-lingual generalization.Aya generalizes from English Jigsaw toxicity data to other languages.Source: I Toxicity Detection Task · p. 112
  • 2%–5%: Aya outperforms Aya Safe, showing that safety-mitigated finetuning reduces toxicity-detection ability.The result highlights a capability trade-off between harmlessness-oriented finetuning and safety-related helpful tasks.Source: I Toxicity Detection Task · p. 112

J.1 Safety Preamble Tuning

  • The final safety preamble adapts critique-and-revise into a one-step classify-and-reject prompt, avoiding multiple model queries for harmful-prompt rejection.Prompt exploration found roleplay and generic harmlessness preambles less consistent, motivating explicit harmfulness attributes in the final prompt.Source: §J.1 · p. 112

J Safety Mitigation

  • Safety mitigation prioritizes over-rejection and restricts distillation data to harmful prompts that should be rejected, although some languages can over-focus on listed harm categories.The final preamble’s harmfulness effects are evaluated on multilingual AdvBench examples for Aya Beta, Aya, and safety-distilled variants.Source: §J.1 · p. 112, Table 24 · p. 113

J.2 Harmful Prompts Data Collection

  • Harmful-prompt data combines AdvBench, its multilingual extension, and selected XSafety categories across Aya’s languages, with train/test splits checked for translation leakage.AdvBench uses 400 training and 120 testing examples, while each selected XSafety category uses 160 training and 40 testing examples.Source: §J.2 · p. 113
  • 3%: prompt-generation pairs are removed before finetuning when generations are shorter than 20 characters, longer than 1000 characters, or excessively repetitive.Repetition is filtered when the longest repeated substring exceeds completion length divided by 2.1.Source: §J.2 · p. 113

J.3 Impact of Safety Data Weight … L Model Card

The paper finds that multilingual safety mitigation depends on language-specific distillation weights, while GPT-4 largely agrees with human harmfulness ratings. The Aya model card highlights a 13B-parameter, 101-language open-source model, broad multilingual evaluations, and community-oriented safety research.

J.3 Impact of Safety Data Weight

  • Higher safety-distillation weights generally reduce harmfulness more, with 3% necessary for sufficient mitigation in lower-resource Zulu and Gaelic despite potential benchmark-performance side effects.This extends the safety-performance trade-off with language asymmetry in multilingual settings.Source: §J.3 · p. 114

J.4 LLM Harmfulness Classification Prompt

  • Harmfulness classification asks an evaluator to label whether a response encourages or enables a harmful intent in the target language as “Yes” or “No”.The prompt requires outputting only one binary choice.Source: §J.4 · p. 114

J.5 Agreement between Human and GPT-4 Harmfulness Ratings

  • 88–97% agreement, averaging 93%, is reported between GPT-4 and human harmfulness ratings, although minority-label recall is lower and GPT-4 slightly underestimates harmfulness.The lower minority-label agreement is especially noted for English and Arabic safety-mitigated models.Source: Table 25 · p. 114, §J.5 · p. 115
  • Examples compare harmful-prompt generations from Aya Safe variants on English AdvBench.Table 26 presents cropped generations for harmful prompts from the English AdvBench test data.Source: Table 26 · p. 115

K Example Model Generations

  • Example generations compare mT0, mT0x, and Aya across translated Dolly-human prompts, with human and GPT-4 preferences reported.The examples include Russian and Serbian mT0x outputs and truncate generations at 256 tokens.Source: Table 27 · p. 116

L Model Card

  • The Aya model is an open-source, instruction-finetuned 13B-parameter LLM covering 101 languages, trained on 25M samples with TPU v4 and T5x.The model card identifies Cohere For AI as curator and provides its repository.Source: L Model Card · p. 118
  • The model card introduces evaluations spanning 99 languages and eight task types, while documenting safety limitations and encouraging community red-teaming of the open-source model.The evaluation suite covers discriminative, generative, human-preference, and LLM-preference tasks; safety and bias details are referred to Sections 6 and 7.Source: L Model Card · p. 118

Related papers

Loading 2402.07827v1…