Tiny Aya: Bridging Scale and Multilingual Depth

Motivation and scope

• Tiny Aya addresses uneven multilingual progress by seeking strong, balanced performance without model-scale increases that raise participation barriers and limit system adaptability.The report frames data-centric design, balanced evaluation, and realistic training constraints as a practical path toward efficient, adaptable, and inclusive multilingual AI.Source: §1 · pp. 2–3

Model family

• Tiny Aya releases a 3.35B-parameter pretrained model spanning 70 languages, an instruction-tuned global model, and three region-specialized variants sharing a multilingual foundation.The variants reinforce linguistic clusters while preserving broad multilingual coverage and practical deployment efficiency.Source: §1 · p. 2

Training strategy

• Balanced multilingual mixtures, diversity-aware data selection, tokenization, and fused synthetic and human-generated signals help preserve language-specific structure and stability across languages.Augmentation strategies target naturalness and reduce bias toward dominant languages while supporting downstream alignment and specialization.Source: §1 · p. 2

Evaluation

• Tiny Aya is evaluated across translation, understanding, reasoning, open-ended generation, safety, and cultural-awareness tasks, including language confusion and response naturalness.The framework emphasizes completeness and consistency across languages and domains rather than narrow leaderboard comparisons.Source: §1 · p. 3

Results

• 46 of 55 languages show higher translation quality for Tiny Aya Global than Gemma3-4B on WMT24++, while the family advances the state of the art in West Asia and Africa.Tiny Aya Global also achieves a +5 point average margin on mDolly, region-specialized variants improve translation by up to 5.5 ChrF points in South Asia and 1.7 on average in Africa, and MultiJail safety reaches 91.1%.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Tokenizer data mixture

• The tokenizer achieves the lowest or near-lowest tokens-per-character ratios across most scripts, especially Khmer, Telugu, Gujarati, and Ge’ez, outperforming competing tokenizers on underrepresented scripts.It uses balanced weighting across languages and language-family/script buckets, with a 262k vocabulary.Source: §2.1 · pp. 3–4, Figure 3 · p. 5

2.2 Pretraining data mixture

• Pretraining covers 70 languages plus programming-language data, balances low-resource languages by tokenizer groupings, and filters corpora through language, cleaning, deduplication, domain, and quality procedures.A cooldown mixture upsamples high-quality data and adds instruction-style datasets spanning all 70 languages.Source: §2.2 · pp. 4–5

2.3 Posttraining

• Posttraining groups languages into five regionally and linguistically motivated clusters and expands missing-language coverage using translated, transformed, and synthetically completed data.The clusters are Asia Pacific, Africa, South Asia, Europe, and West Asia.Source: §2.3 · p. 5

2.3.1 Synthetic Data Generation Pipeline

• Synthetic augmentation translates complete examples or prompts, then applies naturalness, cultural-adaptation, and difficulty transformations before generating target-language completions when needed.FusioN aggregates candidate responses from Gemma3-27B-It, Command A, and DeepSeek-V3 using a judge model to improve multilingual quality across languages and tasks.Source: §2.3.1 · pp. 6–7

2.3.2 Machine Translation Data

• The machine-translation finetuning set contains 312k parallel documents in 98 languages after cleaning, language identification, quality estimation, difficulty filtering, backtranslation, and reference-based filtering.Difficulty filtering prioritizes challenging examples, while wider language exposure may improve crosslingual alignment beyond pretraining coverage.Source: §2.3.2 · p. 7

2.3.3 Data Mix

• Regional mixtures combine selected language clusters with English and code shared across clusters, but availability makes distributions non-uniform: Europe dominates most clusters, while South Asia has nine focus languages across seven scripts.The South Asian cluster therefore has greater English dominance, and European and West Asian representation is similar across three clusters.Source: §2.3.3 · pp. 7–8, Table 1 · p. 7

3.1 Pretraining Stage

• Tiny Aya uses parallel Transformer blocks, interleaved sliding-window and full attention in a 3:1 ratio, SwiGLU without biases, and grouped-query attention.Sliding-window layers use RoPE, full-attention layers use NoPE, and grouped-query attention reduces inference-time memory footprint.Source: §3.1 · p. 8
• Tiny Aya is pretrained for 6T tokens with WSD scheduling, using 200B-token ablations and 40B-token cooldowns to guide data-mixture and hyperparameter choices.Pretraining uses Fax on 256 Nvidia H100 GPUs with mixed FP8, BF16, and FP32 precision.Source: §3.1 · p. 9

3.2 Posttraining Stage

• Posttraining trains models for 3 epochs with cosine decay, then applies minimal preference tuning to Tiny Aya Global to preserve multilingual safety and prevent catastrophic forgetting.Runs use a peak learning rate of 2.5×10−5, final learning rate of 1.2×10−6, and global batch size 32; posttraining completes within 24 hours on 16 H100 GPUs.Source: §3.2 · p. 9

3.3 Model Merging Stage

• Region-specialized and global checkpoints are merged with SimMerge-selected operators and strengths, then evaluated for regional performance, minimum language performance, and multilingual safety.Candidate merges use Linear interpolation, Slerp, and TIES; shared architecture and tokenizer enable direct parameter-space merging without additional training.Source: §3.3 · p. 10

4.1 Development Priorities

• Development uses diverse regional subsets spanning scripts, language families, and resource levels, with model selection based on average performance across each region’s languages.The suite tracks Global MMLU Lite, multilingual safety, Flores, mDolly, and GlobalMGSM to detect regressions and regional disparities.Source: §4.1 · p. 11

4.2 Benchmark Extensions

• The benchmark program extends existing evaluations through careful translation and aggregation of distributed multilingual resources to improve focus-language coverage.These extensions target both open-ended generation and mathematical reasoning.Source: §4.2 · p. 12

4.2.1 Benchmark Translation

• Translated mDolly and mArenaHard-v2.1 expand open-ended evaluations, with filtering and code-aware retranslation designed to reduce nonsensical prompts and major mistranslations.Translation quality is checked with XCOMET-XL, although coding scores and long-context evaluations can be unreliable.Source: §4.2.1 · p. 12

4.2.2 GlobalMGSM: Aggregation of Distributed MGSM Translations

• GlobalMGSM evaluates mathematical reasoning in 35 languages using revised and distributed translations, zero-shot language-specific CoT prompts, and an answer format tailored to each language.The benchmark is preferred for small models because its broad coverage and revisions outweigh the greater challenge of alternatives such as PolyMath.Source: §4.2.2 · pp. 12–13

4.3 Open-ended LLM Judge Evaluations

• Open-ended evaluations use greedy decoding and GPT4.1 (gpt-4-1-04-2025) as the primary LLM judge, despite known bias and crosslingual-alignment limitations.The judge was selected through preliminary analyses and aligns with findings from the multilingual LLM-as-a-judge shared task at WMT 2025.Source: §4.3 · p. 13

4.3.1 Rubric-based absolute ratings in lieu of win rates

• Rubric-based absolute ratings replace win rates to reduce variance, avoid misleading competitor anchoring, and improve interpretability across languages.Judges score accuracy, instruction following, coherence, and fluency from 1 to 7, average the categories, and map the result linearly to [0, 1].Source: §4.3.1 · pp. 13–14
• Rubric judgments remain biased proxies, but they provide scalable directional feedback; evaluations also measure whether outputs remain in the user’s requested language.Language confusion is tracked because global accessibility depends on matching the user’s language unless another language is requested.Source: §4.3.1 · p. 14

4.3.2 Multilingual Safety Evaluations

• Safety evaluation prioritizes both mean and minimum regional scores to reduce cross-language disparities, using internal development tests plus MultiJail across 10 languages and English XSTest.Command A judges MultiJail responses in contextual safety mode without translating outputs to English.Source: §4.3.2 · p. 14

5.1 Discriminative Tasks

• Tiny Aya performs within the same-scale competitor range on Global MMLU, INCLUDE, and Global PIQA, spanning 42, 44, and 116 languages respectively.Aggregate benchmark results are reported as average accuracy and standard deviation in Table 4, with consistent evaluation through lm-eval and greedy decoding for Global PIQA.Source: §5.1 · p. 15, Table 4 · p. 15

5.2 Generative Tasks

• Tiny Aya shows strong open-ended performance, highest naturalness across mDolly and mArenaHard, more stable quality in lower-web-presence languages, and 39.2% African mathematical-reasoning accuracy.Its African-language reasoning accuracy exceeds Gemma3-4B’s 17.6% and Qwen3-4B’s 6.25%, while incorrect-language outputs are 3%–6% higher for those competitors on average per task.Source: Figure 5 · p. 15, §5.2 · p. 16, Table 5 · p. 16
• Tiny Aya Global scores highest on average on both translation tasks, winning against Gemma3-4B in 46/55 WMT24++ languages and outperforming TranslateGemma-4B on 27/66 Flores focus languages.Region-specialized variants improve English-to-target translation across all regions, with gains from +1.7 ChrF points in Africa to +5.5 in South Asia.Source: §5.2 · p. 17, Figure 6 · p. 17, Figure 7 · p. 18

5.3 Safety

• Tiny Aya Global is the safest evaluated model across MultiJail, maintaining a 94% safe response rate for Swahili while competitors produce up to 91% invalid responses.Safety depends on language-specific SFT exposure, and merging with the Global variant reduces multilingual safety gaps for region-specialized models, especially Fire and Earth.Source: §5.3 · pp. 18–19, Figure 8 · p. 18, Table 7 · p. 19

5.4.1 Social Norm Reasoning Analysis

• On NormAd, a Tiny Aya variant is best in 39 of 75 countries, with strongest gains from Fire and Global in West Asia, Asia-Pacific, and the Americas.Test-time chain-of-thought reasoning consistently improves accuracy in both local-language and English settings, although English prompting remains stronger and Africa shows the largest language-related drop.Source: §5.4.1 · pp. 20–21, Figure 9 · p. 20, Figure 10 · p. 21

5.4.2 Cultural Commonsense Reasoning Analysis

• On BLEnD, Tiny Aya improves over Gemma3-4B in 8 out of 16 regions and remains comparable elsewhere, with Tiny Aya Fire delivering the largest gains in low-resource regions.English prompting generally improves accuracy, especially in South Asia; Tiny Aya Fire is most sensitive, reflecting its high proportion of English training data.Source: §5.4.2 · pp. 21–23, Figure 11 · p. 22, Figure 12 · p. 22

• Tiny Aya supports edge inference through q4_0, q4_k_m, and q8_0 formats with MLX and llama.cpp inference stacks.These are standard, widely supported quantization formats and inference stacks.Source: §6 · p. 23
• 32 tokens/s decoding on iPhone 17 Pro versus approximately 10 tokens/s on four-year-old iPhone 13 demonstrates practical on-device inference.Without quantization, the model quickly runs out of memory even on newer devices; older hardware also has lower prefill throughput and higher TTFT.Source: §6 · p. 23
• Quantization penalties generally increase as language web presence decreases, but taper in the lowest-presence bins, indicating robustness in data-scarce languages.The analysis uses mDolly judge-score degradation relative to unquantized BF16 baselines across five Common Crawl-based web-presence bins.Source: §6 · p. 24
• 2.14 GB memory, 32.4 tokens/s throughput, and 1.4-point degradation make Q4_K_M the optimal quantization scheme.Q8_0 has negligible average degradation, while Q4_0 degrades by 2.1 points.Source: §6 · pp. 23–24, Figure 14 · p. 24

The curse of multilinguality

• The curse of multilinguality causes per-language performance drops through capacity constraints and cross-language interference, with transfer mainly benefiting related languages.Lower-resource languages depend more heavily on crosslingual transfer because their training data are typically scarce; proposed remedies include synthetic data, translation, data-mixture weighting, and distillation.Source: §7 · p. 24

Compact multilingual LLMs

• Compact multilingual models trade coverage against capacity: Qwen3-4B and Gemma3-4B cover over 100 languages, whereas SmolLM3 focuses on six high-resource languages and remains competitive with larger-coverage 3B–4B models.Qwen3-4B evaluates only 65 of its 119 training languages, while SmolLM3 uses synthetic reasoning traces for selected SFT domains.Source: §7 · pp. 24–25

Region and task-specific LLMs

• Region- and task-specific models address multilinguality by narrowing scope, including SEA-LION for 11 Southeast Asian languages, EuroLLM-9B for 35 European languages, and Tower for translation across 10 languages.These models use continued pretraining or training from scratch, filtering, synthetic data, and task-specific instruction tuning to improve regional or translation performance.Source: §7 · pp. 25–26

Massively multilingual LLMs

• Apertus extends massively multilingual training beyond 1,000 languages using a 15T-token corpus, but its benchmark performance and evaluation coverage remain limited by the curse of multilinguality and few benchmarks exceeding 100 languages.Approximately 40% of its pretraining data is non-English, emphasizing diverse languages.Source: §7 · p. 26

Tiny Aya

• Tiny Aya trains 3.35B-parameter base, global, and regional models on 70+ languages, outperforming leading models in translation while matching them on multilingual generation.The regional variants are Tiny Aya Water, Tiny Aya Earth, and Tiny Aya Fire.Source: §7 · p. 26

Conclusion

• Tiny Aya demonstrates competitive and stable performance across 70 languages while remaining practically deployable, showing that multilingual capability need not scale with parameter count.The model’s results are attributed to deliberate data mixture design, principled merging, and region-aware specialization.Source: §8 · p. 26
• Balanced multilingual performance is presented as a consequence of intentional design, including data curation, capacity allocation, and structured specialization, rather than monolithic model growth.The conclusion identifies data mixture balancing and cluster-aware training as extensible beyond geography to linguistic structure, domain, or modality while preserving crosslingual transfer.Source: §8 · p. 26
• Future multilingual evaluation should prioritize performance variance and minimum performance across languages, aligning improvements with practical deployment rather than leaderboard averages.The proposed direction targets multilingual models that are efficient, extensible, safe, and globally representative.Source: §8 · p. 26

A Language Distribution Details by Training Region

• The appendix reports English, European, West Asian, South Asian, Asia Pacific, and African language data proportions across data mixes.Tables 8–13 organize these distributions by language region.Source: Table 8 · p. 39, Table 9 · p. 39, Table 10 · p. 39

B.1 mArenaHard Revision

• mArenaHard revision prompts instruct the system to extract natural-language question text while ignoring code, comments, and technical syntax.The template also requires preserving the original text exactly and returning no explanation.Source: §B.1 · p. 41

B.2 GlobalMGSM

• GlobalMGSM prompts require reasoning before a final integer answer formatted with an answer keyword on the last line.The template prohibits adding anything beyond the integer after the answer keyword.Source: §B.2 · p. 41

B.3 LLM Judge Prompt

• The open-ended LLM judge scores instruction following, naturalness, coherence, and accuracy on a 1–7 Likert scale with a concise rationale.The judge prompt specifies that the rationale should be under 50 words and identify the most critical error when applicable.Source: §B.3 · p. 42
• The LLM judge rubric defines score levels for instruction following, naturalness, coherence, and accuracy, ranging from severe failure at 1 to strong performance at 7.The criteria cover adherence to instructions, fluent language, logical structure, and factual correctness.Source: §B.3 · p. 42

C MultiJail Safety Judge Prompt

• MultiJail evaluates response safety independently of the harmfulness of the query and classifies outputs as safe, unsafe, or invalid.Safe responses avoid unsafe content or refuse harmful requests; invalid responses are unnatural or irrelevant.Source: C MultiJail Safety Judge Prompt · p. 43

D Translation Templates

• WMT24++ translation prompts request culturally appropriate, target-language-only translations, while Flores prompts require only the translation; TranslateGemma is queried directly instead.The WMT24++ template specifies source and target languages and target country, whereas Flores uses a shorter translation-only instruction.Source: D Translation Templates · p. 43

E Cultural Benchmarks

• NormAd evaluates stories written in English and in each country’s official language, with per-region accuracy reported separately.Tables 16 and 17 contain the full results for these two story-language settings.Source: E Cultural Benchmarks · p. 44, Table 16 · p. 44, Table 17 · p. 44
• BLEnD organizes evaluation by country, region, and official language, reporting results for source-language and English prompts.Tables 18–20 define the regional language assignments and provide the full evaluation results.Source: E Cultural Benchmarks · p. 44
• NormAd maps each country to an official language for multilingual translation evaluation.Table 15 provides the country-to-language mapping.Source: Table 15 · p. 44

F Generative benchmark results by language

• The generative benchmark section organizes BLEnD countries and languages by geographic grouping.This scope is presented in Table 18.Source: Table 18 · p. 45
• The section combines geographic language coverage, BLEnD question-answering accuracy, and Flores translation-quality evaluation.These components are represented by Tables 18–20 and the Flores comparison in Table 21.Source: Table 18 · p. 45, Table 19 · p. 45, Table 20 · p. 45
• Table 19 reports BLEnD short-question-answer accuracy with source-language prompts.The supplied caption specifies the BLEnD SQA split and source-language prompting condition.Source: Table 19 · p. 45
• BLEnD evaluation in this section covers both language-conditioned and English-only prompting settings for short question answering.The two settings are represented separately in Tables 19 and 20.Source: Table 19 · p. 45, Table 20 · p. 45
• Table 20 reports BLEnD short-question-answer accuracy with English-only prompts.This provides a distinct prompting condition from the source-language evaluation.Source: Table 20 · p. 45

F.1 Translation

• Table 21 compares Gemma and Tiny Aya Global on Flores translation quality using ChrF for English-to-focus-language translation.The supplied passage identifies the comparison, benchmark, metric, and translation direction but does not provide cell values.Source: §F.1 · p. 46

F.2 Mathematical Reasoning

• GlobalMGSM results are reported by language for mathematical reasoning.The results are organized by language in Table 23.Source: §F.2 · p. 46
• The mathematical-reasoning evaluation reports task accuracy for each language.Table 23 includes task accuracy as one of its reported measures.Source: §F.2 · p. 46
• The evaluation also reports language pass rate, defined as whether the output is in the prompt language.This language-specific measure accompanies task accuracy in Table 23.Source: §F.2 · p. 46

F.3 Open-ended Generation

• Open-ended-generation results are presented individually by language.The language-specific results are listed in Tables 24 and 25.Source: §F.3 · p. 46
• Table 24 provides language-level results for open-ended generation.The passage identifies Table 24 as one source of the individual results.Source: §F.3 · p. 46
• Table 25 provides an additional set of language-level open-ended-generation results.The passage identifies Table 25 alongside Table 24 as listing individual results by language.Source: §F.3 · p. 46

Safety evaluation

• MultiJail evaluates Tiny Aya’s multilingual safety, reporting the rates of safe and invalid responses.The evaluation uses Command A as a judge in contextual safety mode and provides detailed by-language scores.Source: G Safety · p. 46, Table 26 · p. 50