Source-linked AI summary
Aya 23: Open Weight Releases to Further Multilingual Progress
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
TL;DR
Multilingual progress remains uneven because robust pretrained models and diverse instruction data are scarce, while broad multilingual models must divide capacity across many languages. Aya 23 pairs Command-based pretrained models with the Aya multilingual instruction collection for 23 languages, and reports stronger benchmark and preference results than Aya 101 and comparable open-weight baselines. Its released 8B and 35B weights improve access to multilingual modeling, but coverage remains a small fraction of global linguistic diversity.
Problem
Robust multilingual pretrained models and diverse multilingual instruction data remain scarce, leaving progress uneven relative to languages such as English and Chinese.
Method
Aya 23 combines Cohere Command-based pretrained models with the Aya multilingual instruction-style collection, focusing capacity on 23 languages included during pretraining.
Results
Aya-23-35B achieves the highest results across covered evaluation tasks and languages, while Aya-23-8B shows best-in-class multilingual performance and reports gains of up to 41.6% on multilingual MMLU over Aya 101.
Takeaways & Limitations
Releasing open weights for 8B and 35B models expands access to multilingual modeling for the 23 supported languages.
Takeaways & Limitations
The 23-language subset captures only a tiny fraction of the world’s linguistic diversity, so further work is needed to improve coverage and performance together.
Abstract
from arXiv · showhide
This technical report introduces Aya 23, a family of multilingual language models. Aya 23 builds on the recent release of the Aya model (Üstün et al., 2024), focusing on pairing a highly performant pre-trained model with the recently released Aya collection (Singh et al., 2024). The result is a powerful multilingual large language model serving 23 languages, expanding state-of-art language modeling capabilities to approximately half of the world's population. The Aya model covered 101 languages whereas Aya 23 is an experiment in depth vs breadth, exploring the impact of allocating more capacity to fewer languages that are included during pre-training. Aya 23 outperforms both previous massively multilingual models like Aya 101 for the languages it covers, as well as widely used models like Gemma, Mistral and Mixtral on an extensive range of discriminative and generative tasks. We release the open weights for both the 8B and 35B models as part of our continued commitment for expanding access to multilingual progress.
1 Introduction
Aya 23 introduces open-weight multilingual instruction-tuned models that focus greater capacity on 23 pre-trained languages rather than Aya 101’s broader 101-language coverage. Across the covered languages, Aya 23 reports stronger results than Aya 101 and comparable open-weight models.
- Aya 23 serves 23 languages with two open-weight sizes, 8B and 35B parameters, building on Cohere’s Command model and the Aya instruction-style collection.
- Allocating more capacity to 23 pre-trained languages addresses the breadth–depth trade-off associated with massively multilingual models such as Aya 101.
- Aya-23-35B achieves the highest results across all covered evaluation tasks and languages, while Aya-23-8B delivers best-in-class multilingual performance among smaller models.
- Up to 14% gains on discriminative tasks, 20% on generative tasks, and 41.6% on multilingual MMLU are reported relative to Aya 101.
- The evaluation covers multilingual benchmarks and human or LLM-judge comparisons against Aya 101, Gemma, Mistral, and Mixtral.
2 Pre-trained Models
Aya 23 uses Cohere Command-series decoder-only Transformer base models, with architectural choices targeting efficiency, stability, long-context behavior, and downstream performance. The family includes distinct parameter configurations for its 8B and 35B models.
- Aya 23 is based on Cohere Command-series models pretrained on a 23-language data mixture, with Aya-23-35B further fine-tuned from Command R.
- The models use a standard decoder-only Transformer with parallel attention and feed-forward layers, SwiGLU activations, and no biases in dense layers.
- RoPE positional embeddings support long-context extrapolation and improve short-context downstream performance relative to ALiBi.
- A 256k BPE tokenizer uses NFC normalization, splits digits into individual tokens, and is balanced across languages for efficient representations.
- Aya-23-8B uses grouped-query attention to reduce inference-time memory footprint by sharing key-value heads across multiple query heads.
- Training uses Fax on TPU v4 chips with data and model parallel submeshes to shard parameters and optimizer states while maintaining throughput.
3 Instruction Fine-Tuning
Aya 23 instruction fine-tuning combines multilingual templates, human annotations, translations, and synthetic data across the 23 target languages. The models are trained with packed long-context examples and role-aware chat formatting.
- 3.1 Data mixture: The fine-tuning mixture combines curated multilingual instruction data with templates, human annotations, translated datasets, and synthetic responses.
- 3.1 Data mixture: Multilingual templates provide 55.7M examples spanning 23 languages and 161 datasets.
- 3.1 Data mixture: Filtering the Aya collection for the training languages yields 55K human-curated prompt-response samples from native speakers.
- 3.1 Data mixture: Translated instruction data is sampled by language and dataset to preserve instance-level diversity, while Command R+ generates 1.63M multilingual synthetic examples.
- 3.1 Data mixture: The fine-tuning mix emphasizes supervised datasets with self-reported commercially permissive licenses and provenance filtering.
- 3.2 Training details: Instruction fine-tuning runs for 13,200 update steps with an 8192-token context, packed data, approximately 10.5M training samples, and batch size 64.
- 3.2 Training details: Special chat-format tokens mark user and chatbot roles and turn boundaries; omitting the format reduces generation quality, although the model remains single-turn rather than explicitly chat-optimized.
4 Multilingual Evaluation
Aya 23 is evaluated across unseen discriminative tasks, multilingual understanding, mathematical reasoning, generative tasks, preference judgments, safety, toxicity, and bias. The evaluation compares Aya 23 with multiple multilingual and widely used open-source baselines across varied model sizes and training regimes.
- The evaluation spans unseen discriminative tasks, multilingual MMLU, multilingual mathematical reasoning, translation, summarization, preference judgments, and safety-related measures.
- Discriminative evaluations use zero-shot scoring on all available dataset languages, while multilingual MMLU uses translated questions and five-shot evaluation.
- XNLI is omitted because all evaluated models performed poorly under the eval-harness setup, which the authors relate to different prompting.
- Aya 23 is assessed with both LLM-simulated win rates and human preference judgments on multilingual Dolly-based test sets.
- The comparison set covers open-source models ranging from 7B to 46B parameters, including mT5, Llama, Gemma, and Mistral-based systems.
5 Results
Aya 23 achieves strong multilingual performance across discriminative, mathematical, generative, preference-based, and safety evaluations. Results support combining a stronger pretrained base model with focused coverage of 23 languages.
- 5.1 Discriminative Tasks: 70.8% average accuracy lets Aya-23-35B outperform all baselines on zero-shot discriminative tasks, including Mixtral-8x7B-Instruct-v0.1 at 68.8%.
- 5.1.1 Multilingual MMLU: Aya-23-8B averages 48.2% accuracy and ranks first in 11 of 14 languages among comparable smaller models, while Aya-23-35B beats Mixtral-8x7B 58.2% to 57.1%.
- 5.2 Multilingual Mathematical Reasoning: Aya 23 improves multilingual mathematical reasoning, with Aya-23-8B scoring 36.6 versus Aya-101-13B's 8.1 and Aya-23-35B scoring 53.7 versus Mixtral-8x7B's 50.2.
- 5.3 Generative Tasks: Aya-23-35B outperforms Mixtral-8x7B by 7.8 spBleu in translation and 23.8 RougeL points in summarization, while Aya-23-8B exceeds Aya-101-13B by 4 spBleu in translation.
- 5.4 Simulated Win Rates and Human Eval: GPT-4 evaluation favors Aya-23-8B over Aya-101-13B in 82.4% of comparisons, and human evaluation gives Aya-23-8B and Aya-23-35B win rates of 50.8% and 57.6%.
- 5.5 Safety, Toxicity & Bias: Harmful responses are lower for Aya 23 than Aya-101-13B in all evaluated languages, with the average rate reduced by at least half, although targeted multilingual safety alignment was not performed.
6 Conclusion
Aya 23 advances multilingual language technology through strong performance across 23 languages and open-weight releases, while acknowledging substantial coverage limitations. The authors identify underrepresented languages and broader linguistic diversity as priorities for future work.
- Aya 23 achieves high performance across a broad range of multilingual benchmarks and human evaluations while releasing its model weights.
- The 23-language coverage remains only a tiny fraction of the world’s approximately 7,000 languages, requiring simultaneous improvement in coverage and performance.
- Aya 23’s language coverage is limited to languages present during pre-training and underrepresents languages spoken in Asia and Africa.
- Future work will target remaining languages through improved coverage, data collection, representation, and attention to cultural and linguistic nuances.
A Languages in Aya 23 Model Family
Aya 23 supports 23 languages, with the accompanying table organizing them by script, language family, subgroup, and approximate native-speaker population.
- The table lists each supported language alongside its script, family, subgrouping, and approximate number of native speakers.