PaperlayerView exact revision on arXiv

Source-linked AI summary

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, Sara Hooker

arXiv:2402.07827v1cs.CL

TL;DR

Aya addresses the concentration of language-model progress in data-rich languages by building an open multilingual instruction-finetuned model. It combines broad and curated multilingual training data with expanded evaluation and reports strong performance across automatic and human settings while covering 101 languages.

  • Problem

    Recent NLP breakthroughs and instruction-following capabilities have centered on a few data-rich languages, while comprehensive multilingual instruction data and evaluation remain limited.

  • Method

    Aya finetunes a 13B-parameter mT5 model using a weighted mixture of multilingual templates, human annotations, translations, and synthetic data, with expanded evaluation and safety analyses.

  • Results

    Aya vastly improves over massively multilingual open-source models across automatic and human evaluations while covering 101 languages and evaluating 99 languages.

  • Takeaways & Limitations

    The work releases multilingual model and data resources and broadens what representative multilingual model development and evaluation can encompass.

  • Takeaways & Limitations

    Translation expands coverage but introduces translationese, and its trade-off with multilingual instruction-following performance remains incompletely understood.

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101

1 Introduction

Aya addresses linguistic inequality in language-model development by combining broad multilingual instruction finetuning with expanded evaluation and open release. It covers 101 languages, including many lower-resourced languages, and reports gains over established multilingual baselines.

  • Development: Aya combines data weighting, pruning, multilingual safety mitigation, and evaluation across varied tasks as part of its development.The work investigates translated, templated, and human-annotated data sources and reports safety reductions for harmful generations.
  • Motivation: Aya targets multilingual instruction following directly in users’ included languages rather than requiring prompts in English.The stated goal is downstream performance when prompts are written in any included language.
  • Language coverage: 101 languages, including 51 lower-resourced languages, are covered by Aya’s instruction-fineting mixture.The mixture contains 203M data points and is 21.5% English, compared with 39% English in xP3.
  • Evaluation: 13.1% and 11.7% relative gains over mT0x are reported for discriminative and generative tasks, respectively.Human preference evaluations across seven languages report a 75% win rate relative to mT0x.
  • Release: The model and instruction datasets are released under permissive open-source terms to support multilingual research and applications.The Aya model is available under an Apache 2.0 license.

2 Data

Aya constructs a broad multilingual instruction-training mixture from curated, human-annotated, translated, and synthetic data. The pipeline emphasizes provenance, quality, diversity, and representation of lower-resourced languages while acknowledging translation-related trade-offs.

  • Language resourcefulness: Language resourcefulness is categorized using recorded, written, and catalogued NLP resources as a proxy for data availability.The 101-language mixture is split into 24 higher-, 26 mid-, and 51 lower-resourced languages.
  • Multilingual templates: xP3x expands xP3 from 86M examples across 46 languages and 13 tasks to 680M examples across 277 languages and 16 tasks before pruning.Aya uses a pruned subset containing 168M examples across 101 languages and 56 datasets.
  • Pruning: Human reviewers remove low-quality, duplicated, and grammatically or structurally flawed templates from xP3x.At least two reviewers inspect every template, with a third resolving disagreements.
  • Pruning: 50.2% of English and 35.9% of multilingual templates are removed, while average instruction length rises 7.0% and 16.8%, respectively.Instance counts decrease by 19.7% for English and 18.3% for multilingual data.
  • Human annotations: The Aya dataset adds natural multilingual annotations collected through a participatory initiative involving 2,997 participants from 110 countries.The project aims to provide organic examples from fluent speakers rather than relying only on automatic or templated construction.
  • Automatic translation: Translation expands coverage but introduces translationese, whose trade-off with multilingual instruction-following performance remains incompletely understood.The paper provides early guidance through an ablation experiment.

3 Experimental Set-up

Aya is trained by finetuning a 13B-parameter mT5 model on a weighted multilingual mixture and is compared with open-source baselines spanning different model sizes, architectures, and language coverage.

  • Base model: Aya finetunes the 13B-parameter mT5 model, whose pretraining covers 101 languages.mT5 is an encoder-decoder transformer pretrained on 1 trillion tokens.
  • Sampling: The 25M-sample training budget motivates source-level and dataset-level sampling to balance task, language, coverage, and data-quality considerations.Weights can prioritize human-annotated, templated, translated, or particular datasets.
  • Sampling ablations: The weighting ablations compare Human Annotation Heavy, Translation heavy, and Template heavy mixtures.These schemes respectively upweight the Aya Dataset, translated sources, or the Aya Collection, xP3x, and Data Provenance.
  • Baselines: Baselines span 13B to 176B parameters, multiple base architectures, language coverages, and training regimes.The comparison includes mT0, BLOOMZ, mT0x, Bactrian-X, and Okapi.
  • Baselines: mT0x extends the xP3 training setup to 101 languages without downsampling or other filtering, providing a coverage-matched comparison.Its shared mT5 base helps isolate differences associated with the Aya instruction-training mixture.
  • Evaluation design: Evaluation distinguishes unseen tasks excluded from training from seen generative tasks evaluated on held-out validation and test splits.Table 4 also treats multilingual MMLU as an unseen dataset.

4 Evaluation

The evaluation suite broadens multilingual assessment beyond standard unseen-task benchmarks by covering discriminative, generative, open-ended, human, and simulated-judge settings. Its datasets collectively extend evaluation to 99 languages, although many individual tasks remain narrower.

  • Evaluation axes: The suite includes unseen discriminative tasks, multilingual MMLU, generative tasks, human evaluation, and simulated win-rate comparisons.The design addresses both held-out task performance and broader open-ended generation capabilities.
  • Coverage: 99 of Aya’s 101 training languages are covered by the expanded evaluation suite.Only Frisian and Latin are excluded, while many individual tasks still cover 10–15 languages.
  • Discriminative tasks: Unseen-task evaluation uses XWinograd, XNLI, XCOPA, and XStoryCloze across coreference resolution, sentence completion, and natural language inference.Holding these tasks out enables direct comparison with mT0 and BLOOMZ.
  • Generative tasks: Generative evaluation covers translation, summarization, and question answering with FLORES-200, XLSum, and TydiQA GoldP.FLORES-200 and XLSum provide coverage across 99 languages, including lower-resourced languages.
  • Human evaluation: Human evaluation uses compensated professional annotators to compare model completions, allowing ties but discouraging them.The evaluation covers seven languages using human-edited multilingual and original English prompts.
  • Simulated win rates: GPT-4 serves as a proxy judge for pairwise win rates across ten languages spanning higher-, mid-, and lower-resource categories.The machine-translated Dolly test set contains 200 held-out prompts.

5 Results

Aya generally outperforms multilingual baselines across discriminative, generative, multilingual MMLU, and preference-based evaluations while covering 101 languages. Results also show that performance depends on data-mixture weighting, language resourcedness, model capacity, and evaluation limitations.

  • Unseen discriminative tasks: 14.8% improvement over mT0x across 101 languages indicates that data composition, rather than model capacity alone, explains much of the multilingual performance gap.The analysis attributes the difference to pruning oversampled datasets and adding a broader variety of multilingual datasets.
  • Data weighting and generation quality: Model variant rankings depend on task: template-heavy performs best on discriminative tasks and limited-coverage generation, whereas translation-heavy leads on high-coverage translation.The final Aya model uses translation-heavy weights, and open-ended generations show reported errors including repetition, hallucination, grammatical mistakes, and unnatural structures across languages.
  • Generative tasks: 40.8% average spBLEU improvement over mT0x on FLORES-200 is larger than Aya’s gains on XLSum and Tydi-QA.Aya’s relative gains are 1.8% in RougeLsum and 2.2% in F1 on those more limited-coverage tasks; translation direction also matters, with gains of 45.3% for X →English and 34.9% for English →X.
  • Human evaluation: 77% average human preference over mT0 and mT0x confirms that Aya generations are generally preferred across languages.Simulated GPT-4 evaluations report higher win rates, but may favor Aya partly because its generations are generally longer; human raters also observed severe errors and hallucinations in some cases.

6 Safety Mitigation

Aya’s multilingual safety work combines translated safety preambles with safety context distillation to reduce harmful responses across languages. The mitigation substantially lowers adversarial harmfulness, while refusal quality and general-task performance remain imperfect.

  • Preamble baseline: 88% of multilingual AdvBench requests were rejected on average after adding a translated safety preamble.Refusal rates were 72% for Scottish Gaelic and 77% for Hindi, the lowest reported languages.
  • Approach: Multilingual safety context distillation teaches Aya when refusals are appropriate without requiring an explicit safety preamble.The approach uses translated harmful prompts and synthetic refusal data to extend safety mitigation across languages.
  • Unmitigated safety: 89–90% of adversarial prompts produced harmful outputs from unmitigated Aya across Arabic, English, and Hindi.GPT-4 estimates were 7–8 percentage points lower in those languages, with greater divergence in Zulu and Scottish Gaelic.
  • Distillation results: 5% average harmfulness for Aya Safe was lower than the teacher model’s 12% average with a preamble.Human ratings placed Aya Safe between 4–11% harmfulness, while GPT-4 ratings ranged from 1% to 10% across evaluated languages.
  • Remaining issues: Aya Safe’s refusal messages were often apologetic, repetitive, and nonspecific despite preventing harmful responses in almost all cases.The authors identify style, diversity, and conciseness as remaining improvement areas.
  • Trade-offs: Safety context distillation caused 0.2–3.2-point losses on general benchmark evaluations.The authors suspect repetitive, narrow distilled responses reduced the quality of the instruction-finetuning mixture.

7 Benchmarking Toxicity and Bias

Aya’s toxicity and bias evaluation spans multiple languages, metrics, and demographic settings. The models generally show lower toxicity and higher translation accuracy than mT0x, but gender and evaluator biases remain visible.

  • Evaluation scope: 18 languages across five language families are covered in Aya’s toxicity and bias analysis.The evaluation compares Aya and Aya Safe with mT0x across open-ended toxicity and gender-bias translation tasks.
  • Toxicity methodology: 25 outputs were sampled per non-toxic prompt and language to estimate toxicity across languages.Toxicity was scored with Perspective API using expected maximum toxicity and toxicity probability.
  • Open-ended toxicity: Aya and Aya Safe have lower expected maximum toxicity and toxicity probability than mT0x across all 14 tested languages.The two Aya models have comparable toxicity levels in this open-ended generation analysis.
  • Evaluation caveat: Perspective API may rate German content as more toxic than comparable English and Italian content, complicating cross-language comparisons.The authors caution that higher German and Portuguese toxicity probabilities may partly reflect evaluator bias.
  • Demographic toxicity: Aya and Aya Safe generally produce less toxic generations than mT0x outside English, but Aya is more toxic for several English racial identity groups.The higher English toxicity appears for Asian people, White men, and Indian men.
  • Gender bias in translation: Aya translations have higher overall accuracy than mT0x on average despite male-gender and pro-stereotypical tendencies.Aya Safe shows promising signs for reducing gender disparity while retaining translation accuracy.

8 Related Work

Related work situates Aya within efforts to expand multilingual pretraining, instruction tuning, translation augmentation, dataset weighting, and multilingual toxicity and bias evaluation. These strands motivate broader language coverage and more comprehensive assessment.

  • Pretrained multilingual models: Multilingual pretraining has expanded through models based on corpora such as mC4, with coverage of around 100 languages in significant amounts.mT5 and umT5 are identified among the largest publicly available multilingual pretrained models.
  • Instruction tuning: Instruction tuning evolved from single-task multilingual augmentation toward finetuning on large collections of tasks.This shift frames instruction following as a broad capability rather than a narrowly task-specific adaptation.
  • Translation augmentation: Translation augmentation methods include translate-train, translate-test, language pivots, and back translation.Aya instead translates training datasets into target languages without translating them back.
  • Dataset weighting: Dataset weighting research has explored dynamic selection and weighting across domains and multilingual data.Prior methods target more efficient or better-performing pretraining and finetuning mixtures.
  • Toxicity and bias evaluation: Existing toxicity and bias evaluations typically cover one language or a small language set, while toxicity research remains concentrated on English.This motivates broader multilingual evaluation of model risks.
  • Multilingual LLM safety: Multilingual safety research shows that malicious prompts in lower-resource languages can bypass safeguards in commercial pretrained LLMs.This literature motivates evaluating unsafe responses beyond English.

9 Discussion

The discussion frames Aya as a broad-coverage multilingual model whose deployment involves substantial capacity, coverage, cultural, linguistic, and safety boundaries. The authors emphasize that these boundaries require further research rather than treating current results as complete.

  • Model size: Aya uses 13 billion parameters, limiting usability on many consumer-grade hardware devices.The authors note that compression may reduce computational cost, but the multilingual performance–cost trade-off remains unresolved.
  • Language coverage: Aya’s 101 languages represent only a tiny fraction of the world’s linguistic diversity.The discussion notes that approximately 7,000 languages exist and that 93% are outside the model’s coverage.
  • Cultural representation: Aya’s translated training data overrepresents Global North and Western datasets, potentially narrowing cultural viewpoints.The human-annotated dataset also contains regional annotator skew for some languages.
  • Model behavior: Gender-neutral responses are difficult in languages with explicitly gendered pronouns or required choices about formality.The authors specifically discuss languages such as Arabic, French, Japanese, Indonesian, Javanese, Yoruba, Spanish, and German.
  • Safety scope: Safety context distillation addresses malicious-prompt refusals but covers only one dimension of Aya’s risk profile.The safety analysis reports limited effects on toxicity in open-ended generations.
  • Evaluation coverage: Multilingual toxicity and bias analysis remains limited mainly to mid- and higher-resourced languages.Gender biases in lower-resourced languages may therefore remain outside the current analysis.

10 A Participatory Approach to Research

Aya emerged from a broad cross-institutional collaboration, contrasting with the narrow institutional and regional networks that have characterized much NLP research. The initiative links global participation with open-science goals.

  • Recent NLP breakthroughs have predominantly relied on small, specialized collaboration networks involving researchers from a handful of institutions and regions.
  • The Aya model was made possible by a broad cross-institutional, global collaboration.
  • Aya is presented as an open-science community initiative connected to earlier collaborative efforts such as the BigScience Workshop.

11 Conclusion

The Aya Initiative broadens representation in language modeling through global collaboration, coverage of 101 languages, and openly released resources. The model is reported to improve substantially across multilingual evaluation settings.

  • Over 3000 collaborators from 110 countries contributed to an initiative spanning 101 languages and more than doubling prior instruction-finetuning, evaluation, and safety coverage.
  • Aya vastly improves over massively multilingual open-source models across automatic and human evaluation settings.
  • The model covers languages spanning higher-, mid-, and lower-resource categories, including Cebuano, Welsh, Amharic, Arabic, and many others.
  • The pruning process retained longer, less repetitive, and more coherent instructions, while using four reviewers and discussion-based disagreement resolution.

B.2 List of xP3x Datasets

The xP3x appendix catalogs the datasets and templates considered for Aya, including multilingual, monolingual, code, classification, question-answering, translation, and summarization resources. It also documents language filtering and post-pruning preservation.

  • The catalog spans tasks including code synthesis, topic classification, sentiment analysis, question answering, summarization, translation, and text generation.
  • The listed resources include multilingual datasets such as Flores, MultiEURLEX, TyDiQA-GoldP, MLQA, and GEM/xlsum.
  • Table 12 lists xP3x datasets filtered according to the 101 languages used by the Aya model.
  • The appendix separately records datasets and templates preserved after pruning, including multilingual templates and the ShareGPT Command and Aya Collection subsets.
  • Figure 18 compares the percentage of examples assigned to each language under different weighting schemes.

D Simulated Preference Evaluation

The simulated preference evaluation uses multilingual GPT-4 comparisons alongside human annotation procedures for pairwise preference and harmfulness ratings. The setup randomizes model order, recruits proficient annotators, and reports variable agreement between human and GPT-4 judgments.

  • Evaluation setup: GPT-4 evaluates multilingual pairwise preferences using a human-guideline-based prompt template and randomized model presentation order.
  • Evaluation setup: The evaluation prompt asks GPT-4 to compare two answers for language use, task response, factual and semantic comprehensibility, grammar, and fluency.
  • Human annotation: Human annotators were natively or professionally proficient in the evaluated languages and came from diverse social and employment backgrounds.
  • Human annotation: Each example was rated by one annotator, with 3–4 annotators involved in each task and varying contribution schedules.
  • Agreement and limitations: The majority of examples received annotations from only one annotator, limiting direct evidence about human label variation.
  • Agreement and limitations: Agreement between GPT-4 and human ratings ranged from 38.9% to 86.5%, generally lower for Aya Safe comparisons and higher for mT0 and mT0x.

E.5.2 Agreement between Humans in Pairwise Ratings

Human ratings show fair inter-annotator agreement, while example generations reveal qualitative differences between Aya and mT0(x) across languages.

  • Cohen’s κ averaged 0.38 and agreement averaged 67.4% across repeated human ratings.
  • Humans agreed more with one another than with GPT-4, except for the Aya versus mT0x task in French.
  • Aya generations were more verbose and elaborate, whereas mT0(x) outputs were often shorter, incomplete, or in English for Arabic.

H.2 Co-occurrence Analysis

The co-occurrence analysis examines adjectives and adverbs associated with identity groups, revealing undesirable associations while emphasizing ambiguity in the group labels.

  • Adjectives and adverbs were counted in the first full sentence to reduce noise from words unrelated to the identity group.
  • Identity prompts were grouped into gender, racial, and religion categories, with overlapping words removed from the top-50 comparisons.
  • Aya generations contained undesirable associations such as “evil” and “stupid” for “White man” and “destitute” and “useless” for “Indian man.”
  • Racial identity terms were not geographically disambiguated, making the exact reference of observed subgroup associations difficult to pinpoint.

I Toxicity Detection Task

Aya is evaluated on multilingual toxicity detection using binary classification, with comparisons shaped by training-data exposure and safety finetuning.

  • Toxicity detection classifies text as “toxic” or “non-toxic” using model log-likelihoods for “yes” and “no” continuations.
  • AUC-ROC is the primary metric for English CivilComments and multilingual Jigsaw evaluations.
  • Aya outperformed solely pretrained mT5 models and was described as the best-performing model overall due to cross-lingual generalization.
  • Aya outperformed Aya Safe by 2%–5% in toxicity detection, indicating a trade-off associated with safety-mitigated finetuning.

J.3 Impact of Safety Data Weight

Safety-distillation weight affects harmfulness reduction, while the model card documents Aya’s multilingual scope and evaluation context.

  • Impact of Safety Data Weight: Higher safety-distillation weights generally produced larger reductions in harmfulness, while lower-resourced languages required higher weights.
  • GPT-4 agreed with human harmfulness ratings at 88–97% per task, averaging 93%, but slightly underestimated harmfulness.
  • Model Card: Aya is a 13B-parameter open-source model instruction-finetuned on 101 languages and evaluated across multilingual tasks.
  • Model Card: The model card states that no updates are planned for maintenance.
Loading 2402.07827v1…