Source-linked AI summary
Tiny Aya: Bridging Scale and Multilingual Depth
Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza, Ammar Khairi, David Mora, Saurabh Dash, Viraat Aryabumi, Sara Rajaee, Mehrnaz Mofakhami, Ananya Sahu, Thomas Euyang, Brittawnya Prince, Madeline Smith, Hangyu Lin, Acyr Locatelli, Sara Hooker, Tom Kocmi, Aidan Gomez, Ivan Zhang, Phil Blunsom, Nick Frosst, Joelle Pineau, Beyza Ermis, Ahmet Üstün, Julia Kreutzer, Marzieh Fadaee
TL;DR
Multilingual progress is uneven across languages, while scale-heavy strategies raise barriers to participation and adaptation. Tiny Aya uses balanced data design, multilingual training, and region-aware specialization across a compact 70-language model family. It remains competitive at the same size, improves lower-resource language parity, and achieves strong translation, generation, reasoning, and safety results.
Problem
Multilingual performance remains uneven because gains often track data availability, while scale-heavy improvement strategies raise participation barriers and limit system adaptability.
Method
Tiny Aya combines balanced multilingual data mixtures, diversity-aware training, synthetic-data fusion, broad evaluation, and region-specialized posttraining across a 3.35B-parameter, 70-language model family.
Results
Tiny Aya is competitive with same-size multilingual models, beats Gemma3-4B in translation quality in 46 of 55 WMT24++ languages, and achieves 91.1% mean safe response rate on MultiJail.
Takeaways & Limitations
The results support an efficient alternative to parameter scaling in which deliberate data design and region-aware specialization produce balanced, practically deployable multilingual systems.
Takeaways & Limitations
Rubric-based evaluations remain limited by imperfections and biases in LLM judges, making them useful mainly as a directional proxy for human evaluation.
Abstract
from arXiv · showhide
Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.
1 Introduction
Tiny Aya addresses uneven multilingual progress with a compact model family designed for balanced performance across 70 languages. Its data-centric training and broad evaluation support competitive task performance, reduced language disparities, and practical deployment.
- Multilingual progress remains uneven because performance often follows data availability, while scaling and posttraining increase participation barriers.
- Tiny Aya comprises a 3.35B-parameter pretrained model spanning 70 languages, a globally optimized instruction-tuned model, and three region-specialized variants.
- Its training combines diversity-aware data selection, tokenization research, synthetic and human-generated signals, balanced multilingual mixtures, and region-conscious augmentation.
- Evaluation spans translation, understanding, reasoning, open-ended generation, safety, and cultural awareness while also measuring language confusion and response naturalness.
- Tiny Aya Global beats Gemma3-4B in translation quality in 46 of 55 WMT24++ languages and leads the next competitor by 5 points on average on mDolly.
- The report presents balanced multilingual performance, efficiency, and adaptability as an alternative path toward practical and inclusive language technology.
2 Building a balanced multilingual data mixture
Tiny Aya builds multilingual coverage through balanced tokenization and data mixtures, region-aware clustering, and synthetic augmentation designed to improve naturalness and reduce dominant-language bias.
- Tokenizer data mixture: A single 262k-vocabulary tokenizer covers all 70 languages, enabling language grouping and model merging without vocabulary transfer.The tokenizer is trained on 50GB of Fineweb-2 data using a language- and distribution-aware weighting scheme.
- Tokenizer data mixture: Lower tokens-per-character values indicate better compression, and Tiny Aya is lowest or near-lowest across most scripts, especially Khmer, Telugu, Gujarati, and Ge’ez.The comparison includes Gemma3-4B, Qwen3-4B, and SmolLM3-3B tokenizers.
- Pretraining data mixture: Pretraining data covers 70 languages, includes code, balances low-resource languages, and applies language filtering, cleaning, deduplication, and quality classification.A cooldown mixture further upsamples high-quality and instruction-style data spanning all 70 languages.
- Posttraining data mixture: Posttraining groups languages into five regional clusters based on linguistic, geographic, and resource considerations, extending missing-language coverage through translation, prompt transformations, and synthetic completions.The data construction explicitly balances language coverage, naturalness, and low bias toward dominant languages.
- Synthetic data generation: FusioN combines candidate responses from multiple teacher LLMs through a judge model that extracts and aggregates their strongest components across languages and tasks.Command A serves as the Fusor because of its multilingual performance, safety, inference cost, and crosslingual generalization.
- Regional data composition: English remains the highest-represented language in each cluster, while the European region contributes the largest share except in the South Asian cluster.The South Asian cluster contains nine focus languages across seven scripts, increasing English dominance in its resulting model.
3 Training efficient and adaptable multilingual models
Tiny Aya combines an efficient Transformer architecture with large-scale multilingual pretraining and lightweight, region-aware posttraining. Checkpoint merging is used to retain regional gains while preserving global instruction-following and safety behavior.
- Architecture: Tiny Aya uses a dense decoder-only Transformer with parallel blocks, interleaved sliding-window and full attention, SwiGLU activations, no dense-layer biases, and grouped-query attention.Grouped-query attention reduces inference-time memory footprint, while parallel blocks improve training efficiency without hurting model quality.
- Architecture: Table 2 summarizes Tiny Aya’s key architecture hyperparameters and parameter counts.The supplied passage identifies the table’s scope but does not provide its individual values.
- Pretraining: 6T tokens are used for pretraining with a Warmup-Stable-Decay learning-rate schedule selected through 200B-token smaller-scale ablations.The schedule provides flexibility in determining the pretraining token budget.
- Training infrastructure: Pretraining runs on 256 Nvidia H100 GPUs with FP8, BF16, and FP32 computation, while posttraining cluster models use 16 H100 GPUs and finish within 24 hours.Main weights and optimizer states remain in FP32 during pretraining, with lower-precision casts used for computation.
- Supervised finetuning: All models use three training epochs, cosine learning-rate decay, a 2.5×10−5 peak learning rate, a 1.2×10−6 final learning rate, and global batch size 32.Data mixing and hyperparameters are tuned toward balanced development-set performance.
- Preference tuning: Minimal preference tuning teaches Tiny Aya Global its identity and language list while maintaining multilingual safety and limiting catastrophic forgetting.The lightweight stage is intended to preserve downstream adaptability and personalization.
- Checkpoint merging: Region-specific posttraining can improve cluster-relevant performance but degrade global consistency and safety, so SimMerge selects checkpoint merges to balance both.The released regional models merge region-specialized and global checkpoints sharing the same architecture and tokenizer.
- Checkpoint merging: SimMerge evaluates linear interpolation, Slerp, and TIES merges across mixing strengths, then selects each region’s checkpoint using regional averages, minimum performance, and safety checks.Selection uses task-agnostic similarity features from an unlabeled multilingual probe corpus before merging.
4 Evaluating multilingual capability at scale
Tiny Aya evaluates multilingual capability through a broad, region-conscious benchmark suite spanning translation, generation, reasoning, safety, and language consistency. Its development process emphasizes balanced performance across languages, while using translated benchmark extensions and rubric-based LLM judging as scalable evaluation tools with acknowledged limitations.
- The benchmark suite prioritizes focus-language and regional coverage, generative tasks, complexity, and orthogonality across selected evaluations.
- Development tracks Global MMLU Lite, multilingual safety, and regional Flores, mDolly, and GlobalMGSM subsets across languages differing in script, family, and resource availability.This design checks that improvements are not driven by a single dominant language within a region.
- Model selection uses average performance across development languages for each region, favoring stable multilingual capability over isolated benchmark peaks.Language confusion is also measured to assess whether outputs remain in the user’s requested language.
- The evaluation suite extends existing benchmarks through translation and distributed resources to increase coverage of focus languages, including 35 languages in GlobalMGSM.Translated generative benchmarks are filtered and quality-checked, with coding prompts receiving notably lower XCOMET-XL scores.
- Open-ended evaluation replaces win rates with absolute rubric ratings covering accuracy, instruction following, coherence, and fluency on 1-to-7 Likert scales.The change addresses variance, competitor anchoring, and interpretability problems in multilingual comparisons.
- LLM-judge ratings remain limited by evaluator imperfections and biases, so they are treated as directional, scalable proxies for human evaluation.
5 Results: balanced performance across languages
Tiny Aya remains competitive with same-scale multilingual models while reducing disparities across languages, especially in translation, open-ended generation, African-language reasoning, safety, and regional cultural tasks.
- Discriminative Tasks: Tiny Aya performs within the 3–4B competitor range on Global MMLU, INCLUDE, and Global PIQA, though it does not achieve the highest average scores.These benchmarks cover 42, 44, and 116 languages respectively.
- Generative Tasks: +5 points: Tiny Aya leads the next competitor on average mDolly open-ended generation and has the highest naturalness ratings across open-ended tasks.Competitors show larger cross-language score variation, while Tiny Aya maintains more stable performance in lower-web-presence languages.
- Generative and Reasoning Tasks: 39.2% accuracy: Tiny Aya Global outperforms Gemma3-4B and Qwen3-4B on the African-language subset of mathematical reasoning.The corresponding competitor accuracies are 17.6% and 6.25%.
- Translation: 46 of 55 languages: Tiny Aya Global outperforms Gemma3-4B on WMT24++ translation quality.Tiny Aya Global scores highest on average on both evaluated translation tasks.
- Translation: +5.5 ChrF points: the South Asia region-specialized model improves over Tiny Aya Global on English-to-target translation.Region-specialized models outperform the Global variant across all regions, with a smaller +1.7 ChrF-point boost for Africa.
- Safety: 94% safe responses: Tiny Aya maintains a high safe-response rate for Swahili prompts while reducing multilingual safety disparities.Tiny Aya Global is described as the safest evaluated model, with slightly more over-refusals than competitors.
- Cultural Commonsense Reasoning: 39 out of 75 countries: a Tiny Aya variant outperforms Gemma3-4B on regional performance when prompted in the local language.Tiny Aya also gains over Gemma3-4B in 8 of 16 BLEnD SQA regions, with notable gains in several low-resource-language regions.
- Cultural Commonsense Reasoning: English prompting consistently improves model accuracy, with especially high sensitivity in South Asian regions and for Tiny Aya Fire.This pattern reflects the influence of language coverage and training-mixture balance on multilingual performance.
6 Small, Fast Multilingual AI for Everyone, Everywhere
Tiny Aya is designed for practical multilingual deployment on edge devices, including older phones, using standard quantization formats. Quantization preserves generation quality relatively well, with Q4_K_M offering the strongest reported efficiency-quality trade-off.
- Deployment: Tiny Aya targets on-device multilingual use through standard q4_0, q4_k_m, and q8_0 quantization formats supported by MLX and llama.cpp.These formats are intended to make inference practical on edge devices.
- Deployment: ∼10 tokens per second decoding on an iPhone 13 increased to 32 tokens per second on an iPhone 17 Pro.The standardized workload used 100 input tokens and 100 output tokens; older hardware had lower prefill throughput and higher TTFT.
- Quality under quantization: 1.4 points average mDolly degradation for Q4_K_M and 2.1 points for Q4_0, while Q8_0 showed negligible degradation.Degradation was measured relative to unquantized model generation quality.
- Quality under quantization: Higher-web-presence languages generally show smaller quantization quality deltas, while the penalty increases toward lower web presence before tapering in the lowest bins.The web-presence proxy uses Common Crawl and five equal-width bins; the lowest-presence languages remain comparatively robust.
- Quality under quantization: 2.14 GB memory, 32.4 tokens/s throughput, and 1.4 points degradation make Q4_K_M the reported optimal quantization scheme.The comparison combines memory footprint, throughput, and mDolly quality degradation.
7 Related Work
Related work spans compact, region-specific, task-specific, and massively multilingual models. Tiny Aya occupies the compact massively multilingual space while emphasizing balanced performance across 70+ languages.
- Motivation: Fixed-size multilingual models face a curse of multilinguality: adding languages can reduce per-language performance through capacity constraints and cross-language interference.Crosslingual transfer is reported to benefit related languages more than distant language families.
- Compact multilingual LLMs: Qwen3-4B, Gemma3-4B, and SmolLM3-3B represent compact multilingual alternatives with differing language coverage, training scale, and specialization.SmolLM3-3B limits coverage to six high-resource languages, whereas Qwen3-4B and Gemma3-4B support substantially broader sets.
- Region and task-specific LLMs: Region-specific models such as SEA-LION address multilingual challenges through regional focus, while Tower specializes in machine translation across 10 languages.These approaches trade breadth for regional or task-specific concentration.
- Massively multilingual LLMs: Apertus pushes coverage beyond 1,000 languages, illustrating global breadth while also exposing the scarcity of evaluation suites covering more than 100 languages.Its broad coverage serves as a proof point for global language modeling despite weaker benchmark performance.
- Tiny Aya: Tiny Aya releases 3.35B-parameter Base, Global, Water, Earth, and Fire models trained on 70+ languages, outperforming leading models on translation while matching them on multilingual generation.The family combines broad coverage with compact model size and region-specific variants.
8 Conclusion
Tiny Aya argues that balanced multilingual capability can be achieved through design rather than parameter scale alone. Its approach combines deliberate data mixtures, principled merging, region-aware specialization, and evaluation beyond average performance.
- Conclusion: Tiny Aya demonstrates that multilingual capability does not have to scale with parameter count.The conclusion attributes this result to deliberate data mixture design, principled merging, and region-aware specialization.
- Conclusion: A compact model can deliver competitive and stable performance across 70 languages while remaining practically deployable.The paper frames this as a trade-off between multilingual performance and efficiency.
- Conclusion: Future multilingual progress can emphasize intentional data curation, capacity allocation, and specialization instead of relying on monolithic model growth.The conclusion also suggests extending balancing and cluster-aware training beyond geography to linguistic structure, domain, or modality.
- Conclusion: Evaluation should prioritize variance and minimum performance across languages alongside aggregate capability.This aligns model improvement with balanced multilingual behavior rather than average scores alone.
A Language Distribution Details by Training Region
The appendix provides data-proportion tables organized by training region and code. It also includes prompt templates and evaluator instructions used across several multilingual assessments.
- Regional data tables: The appendix includes separate data-proportion tables for English and European languages across data mixes.The supplied captions identify these as Table 8 and Table 9.
- Regional data tables: Additional tables report data proportions for West Asia, South Asia, Asia Pacific, African languages, and code.These are identified as Tables 10 through 14 in the supplied passages.
- Evaluation materials: The appendix contains templates for open-ended evaluation, mathematical reasoning, safety assessment, and machine translation.The supplied passages include named templates and associated instructions for these evaluation settings.
E Cultural Benchmarks
The cultural-benchmark evaluation maps countries to official languages and reports NormAd accuracy both with English stories and with stories in each country’s official language.
- NormAd evaluation selects an official translation language for each country.
- NormAd reports per-region accuracy when stories are entirely in English.
- NormAd also reports per-region accuracy when stories use each country’s official language.
F Generative benchmark results by language
The multilingual benchmark results organize BLEnD by geographic grouping and prompt language, while Flores evaluates English-to-focus-language translation quality for Tiny Aya Global.
- BLEnD groups countries and languages geographically for evaluation.
- BLEnD short-question-answer results report accuracy with source-language prompts.
- BLEnD short-question-answer results also report accuracy with English-only prompts.
- Flores compares Tiny Aya Global’s English-to-focus-language translation quality using ChrF.
F.2 Mathematical Reasoning
GlobalMGSM reports mathematical-task accuracy alongside whether outputs remain in the prompt language, with results listed separately by language.
- F.2 Mathematical Reasoning: GlobalMGSM reports per-language task accuracy.
- F.2 Mathematical Reasoning: GlobalMGSM also reports language pass rate, measuring whether outputs use the prompt language.
- F.2 Mathematical Reasoning: The mathematical-reasoning results are presented individually by language.
G Safety
The evaluation reports multilingual generation, translation, reasoning, and safety results by language, including response safety and language-consistency measures.
- Translation: Tiny Aya regional models are not recommended for languages outside their training focus because translation quality is significantly lower than Tiny Aya Global’s.
- Translation: WMT24++ evaluates English-source translation quality with ChrF.
- F.2 Mathematical Reasoning: GlobalMGSM reports per-language answer accuracy and language-consistency pass rates based on FastText.
- F.3 Open-ended Generation: Open-ended generation is evaluated on mArenaHard-v2 and mDolly using aggregate and naturalness judge scores plus language-identification pass rates.
- Safety: MultiJail reports the rates of safe and invalid responses by language.