Source-linked AI summary
Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
Hasan Abed Al Kader Hammoud, Mohammad Zbeeb, Bernard Ghanem
TL;DR
Arabic instruction tuning faces a shortage of high-quality data, while language-centric modeling requires faithful translation and Arabic capability across scales. HALA builds a translate-and-tune pipeline to construct Arabic instruction data and train models from 350M to 9B parameters. It reports state-of-the-art average results in the nano and small categories on Arabic benchmarks, outperforming base checkpoints.
Problem
High-quality Arabic instruction data is scarce, constraining instruction tuning and scaling, while the volume and fidelity of existing instruction data lag.
Source: §1 · p. 2Method
HALA compresses an AR↔EN translator to FP8, uses bilingual supervision to train a lightweight translator for translating English instruction sets into Arabic, and fine-tunes and merges Arabic models across scales.
Source: §1 · p. 3, §2.1 · p. 3, §2.4 · p. 4Results
HALA delivers consistent improvements over base Liquid and FANAR checkpoints and achieves state-of-the-art averages in both the nano (≤2B) and small (≤9B) categories on a diverse Arabic benchmark suite.
Source: §4 · p. 7Takeaways & Limitations
The report releases models, data, and recipes to support further Arabic instruction-tuning research and presents language-centric approaches as a complement to breadth-first multilingual scaling.
Source: §4 · p. 7The approach requires reliable translation that preserves instruction fidelity and training strategies that maintain Arabic fluency and task competence across model sizes.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We present Hala, a family of Arabic-centric instruction and translation models built with our translate-and-tune pipeline. We first compress a strong AR$\leftrightarrow$EN teacher to FP8 (yielding $\sim$2$\times$ higher throughput with no quality loss) and use it to create high-fidelity bilingual supervision. A lightweight language model LFM2-1.2B is then fine-tuned on this data and used to translate high-quality English instruction sets into Arabic, producing a million-scale corpus tailored to instruction following. We train Hala models at 350M, 700M, 1.2B, and 9B parameters, and apply slerp merging to balance Arabic specialization with base-model strengths. On Arabic-centric benchmarks, Hala achieves state-of-the-art results within both the "nano" ($\leq$2B) and "small" (7-9B) categories, outperforming their bases. We release models, data, evaluation, and recipes to accelerate research in Arabic NLP.
1 Introduction
The introduction frames Arabic instruction-data scarcity and uneven language coverage as barriers to Arabic-centric capability. HALA responds with an efficient translation-and-tuning pipeline and reports competitive performance in Arabic-centric benchmark size categories.
Arabic LLMs and the instruction-data bottleneck
- High-quality Arabic instruction data remains scarce, constraining instruction tuning and scaling; existing Arabic instruction data still lags in both volume and fidelity.Arabic also presents diglossia, rich morphology, and wide dialectal variation, while available benchmarks have limited coverage and difficulty relative to English.Source: §1 · p. 2
Language-centric vs. multilingual
- The proposed translation-first approach depends on reliable translation that preserves instruction semantics and training strategies that scale while maintaining Arabic fluency and task competence.The report defines language-centric modeling as prioritizing capability depth in Arabic rather than uniform breadth across languages.Source: §1 · pp. 2–3
Our approach and contributions
- The report introduces an Arabic language-centric model family built around an efficient translate-and-tune pipeline.Its contributions include an FP8 AR↔EN translator, million-scale bilingual supervision, a multi-million-pair Arabic instruction corpus, models from 350M to 9B parameters, and open releases.Source: §1 · p. 3
Summary
- HALA models achieve competitive performance within their parameter classes on Arabic-centric benchmarks.The introduction presents this as support for language-centric modeling as a complement to breadth-first multilingual scaling.Source: §1 · p. 3
2 Methodology
The methodology compresses a multilingual translator, builds bilingual and Arabic instruction data, then fine-tunes and merges models across several sizes.
2.1 Quantizing the main translator to FP8
- The multilingual translator is compressed to FP8, reducing memory use and producing approximately 2× faster inference while preserving translation quality on the evaluation sets.The conversion uses dynamic scaling and post-conversion validation.Source: §2.1 · p. 3
2.2 Bootstrapping bilingual supervision from Open-Orca
- 405K Open-Orca instruction–response pairs are translated into Arabic, pairing both questions and responses with their original English counterparts.The resulting bilingual tuples mirror Open-Orca's semantics and difficulty and include substantial coverage of reasoning-heavy queries.Source: §2.2 · pp. 3–4
2.3 Quality filtering of OPUS-100 with a strict bilingual judge
- 439,592 OPUS-100 pairs are retained from approximately 1M candidates after a compact judge model evaluates translation fidelity.The accepted pairs complement instruction-style bilingual data with a clean AR↔EN signal.Source: §2.3 · p. 4
2.4 Training a lightweight AR↔EN translator
- The translated Open-Orca set and filtered OPUS-100 pairs total approximately 1.26M bilingual examples for fine-tuning LFM2-1.2B as an instruction-style AR→EN translator.Training uses chat-style prompting for English-to-Arabic and supervised fine-tuning with cross-entropy; the translator is then used to construct Arabic data at scale.Source: §2.4 · p. 4
2.5 Building the Arabic instruction corpus via translation
- The translated English instruction corpus totals roughly 4.5M samples and emphasizes instruction following, reasoning, and alignment.The source datasets are translated while preserving formatting and answer style.Source: §2.5 · p. 4
2.6 Arabic instruction fine-tunes and model merging
- Models are fine-tuned across scales and merged with their base checkpoints using spherical linear interpolation at t=0.5.The report says this merging was found to preserve general capability while boosting Arabic instruction-following performance.Source: §2.6 · p. 5
3 Evaluation
Evaluation uses six Arabic-centric benchmarks and reports HALA gains over bases and a prior state-of-the-art baseline across nano and small model sizes.
Benchmarks and protocol
- The benchmark suite covers AlGhafa, AraTrust, ArabicMMLU, ArbMMLU-HT, EXAMS, and MadinahQA; Alrage is excluded because it requires an LLM-as-a-judge setup.Evaluations use LightEval with vLLM.Source: §3 · p. 5, Table 1 · p. 6
Model families
- The comparison includes HALA models at 350M, 700M, 1.2B, and 9B alongside LiquidAI checkpoints, FANAR, and representative multilingual and Arabic-centric baselines.The listed model families include LLaMA, Qwen, Gemma, SILMA, Saka, Yehia, ALLAM, and Command-R.Source: §3 · p. 5
Translator quality
- Translation fidelity is assessed on 500 sampled English MMLU questions aligned by subject and item ID to Arabic references, using BLEU, ROUGE-L, and chrF++.Table 2 reports EN→AR quality against the Arabic subset of openai/mmmlu, with deltas against reference systems within each block.Source: §3 · p. 5, Table 2 · p. 6
Prompting setup
- Translation comparisons use fixed prompts, with an MSA-only professional translation prompt for LFM2-1.2B and a generic Arabic translation prompt for other systems.The prompts request Arabic-only output.Source: §3 · pp. 5–6
Main results
- HALA-1.2B has the best average in the nano bucket, HALA-350M and HALA-700M beat Liquid bases across most tasks, and HALA-9B beats QCRI/Fanar-1-9B-Instruct on average.Table 1 reports task scores and unweighted averages across six tasks; higher is better.Source: §3 · p. 5, Table 1 · p. 6
Compute and cost
- Model training used a budget of $1,000 on 8×H100-SXM GPUs, while dataset translation cost roughly $500 on 12×A100 GPUs.Source: §3 · p. 6
4 Conclusion
HALA combines an efficient translate-and-tune pipeline with slerp-based merging and improves over base checkpoints, reaching state-of-the-art averages in the nano and small categories on Arabic benchmarks.
4 Conclusion
- HALA achieves state-of-the-art averages in both the nano (≤2B) and small (≤9B) categories on a diverse Arabic benchmark suite, while consistently improving over base Liquid and FANAR checkpoints.Source: §4 · p. 7
- The pipeline compresses an AR↔EN translator to FP8, bootstraps million-scale Arabic instruction data from high-quality English sources, and fine-tunes compact and small models with slerp-based merging.Source: §4 · p. 7
- The report releases models, data, and recipes to support further Arabic instruction-tuning research and language-centric modeling as a complement to breadth-first multilingual scaling.Source: §4 · p. 7