PaperlayerView exact revision on arXiv

Source-linked AI summary

Fanar: An Arabic-Centric Multimodal Generative AI Platform

Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus'ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, Chaoyi Ruan

arXiv:2501.13944v1cs.CLcs.AI

TL;DR

Arabic-centric generative AI requires broad, clean data and systems that account for Arabic’s linguistic and cultural characteristics. Fanar addresses this gap with two concurrently deployed LLMs, multimodal services, specialized RAG modules, and Arabic-focused preprocessing and training. The platform reports strong Arabic ASR performance and benchmark improvements from selected preference-learning procedures, while identifying data scale and factuality as continuing boundaries.

  • Problem

    Effective Arabic-centric LLM development is constrained by limited clean, broad-coverage Arabic data and by translated content that can introduce English-language cultural contexts.

  • Method

    Fanar combines Fanar Star trained from scratch, Fanar Prime continually trained from Gemma-2 9B, an Arabic morphological tokenizer, multimodal services, and specialized RAG modules.

  • Results

    Fanar ASR consistently outperforms Google ASR and Whisper-Largev2 on LAraBench and SADA, while preference-data batching improves automated benchmarks by +1-2%.

  • Takeaways & Limitations

    Fanar provides a deployed Arabic-centric platform combining concurrent prompt-routed LLMs with speech, image, translation, retrieval, and attribution capabilities.

  • Takeaways & Limitations

    Arabic data remains the biggest bottleneck, and human testers report an approximately 15% dislike rate, mostly due to factuality-related errors.

Abstract

from arXiv · show

We present Fanar, a platform for Arabic-centric multimodal generative AI systems, that supports language, speech and image generation tasks. At the heart of Fanar are Fanar Star and Fanar Prime, two highly capable Arabic Large Language Models (LLMs) that are best in the class on well established benchmarks for similar sized models. Fanar Star is a 7B (billion) parameter model that was trained from scratch on nearly 1 trillion clean and deduplicated Arabic, English and Code tokens. Fanar Prime is a 9B parameter model continually trained on the Gemma-2 9B base model on the same 1 trillion token set. Both models are concurrently deployed and designed to address different types of prompts transparently routed through a custom-built orchestrator. The Fanar platform provides many other capabilities including a customized Islamic Retrieval Augmented Generation (RAG) system for handling religious prompts, a Recency RAG for summarizing information about current or recent events that have occurred after the pre-training data cut-off date. The platform provides additional cognitive capabilities including in-house bilingual speech recognition that supports multiple Arabic dialects, voice and image generation that is fine-tuned to better reflect regional characteristics. Finally, Fanar provides an attribution service that can be used to verify the authenticity of fact based generated content. The design, development, and implementation of Fanar was entirely undertaken at Hamad Bin Khalifa University's Qatar Computing Research Institute (QCRI) and was sponsored by Qatar's Ministry of Communications and Information Technology to enable sovereign AI technology development.

1. Introduction

Fanar is introduced as an Arabic-centric multimodal generative AI platform addressing the challenges of building effective non-English language technologies. Its core models and supporting services combine large multilingual training data with specialized Arabic capabilities.

  • Arabic-focused language technologies are needed because high-quality non-English LLM development remains constrained by limited large datasets.
  • Fanar integrates text-based LLMs, speech and image generation, specialized RAG modules, and attribution services for generated facts.
  • Fanar Star is a 7B-parameter model trained from scratch, while Fanar Prime is a 9B-parameter model continually trained from Gemma-2 9B.
  • Both models are trained on nearly 1 trillion clean and deduplicated Arabic, English, and code tokens, with the models working in concert to address Arabic-data gaps.
  • The report covers Arabic context, platform components, data preparation, tokenizer design, and model architectures before discussing evaluation and applications.

2. Arabic Language

Arabic combines broad cultural and geographic significance with linguistic properties that complicate computational processing. Its morphology, diglossia, script, and limited web representation motivate Arabic-specific language technologies.

  • Arabic is spoken by more than 467 million people across 60 countries and is the liturgical language of over 2 billion Muslims.
  • Arabic spans Classical Arabic, Modern Standard Arabic, and regional colloquial dialects with substantial linguistic differences.
  • Arabic words typically derive from consonantal roots combined with morphological templates, prefixes, and suffixes to express nuanced meanings.
  • Diglossia complicates processing because Modern Standard Arabic serves formal uses while native speakers primarily use regionally varying dialects.
  • Arabic script is written right to left, and omitted short vowels can require diacritization for accurate interpretation.
  • Arabic content accounts for only 0.5% of online data despite the language’s large speaker population and cultural impact.

3. Fanar Platform Services

Fanar coordinates Arabic-centric language, speech, image, translation, and retrieval services through an orchestrator. Its two LLMs are routed by prompt type, while safety filtering and specialized modules support culturally aligned use.

  • The orchestrator coordinates platform services, routing requests from the chat application or API to direct speech and translation services or classified LLM-driven services.
  • Safety and orchestration: Responses and image-generation prompts pass through safety filtering to support responsible AI guidelines and cultural alignment.
  • Language models: Fanar Star is a 7B-parameter model trained from scratch with a two-stage curriculum over Arabic, English, and code data.
  • Language models: Prompts outside Islamic or STEM domains are routed to Fanar Star, while Fanar Prime handles STEM and reasoning-related prompts.
  • Speech and generation: Fanar’s Arabic-English speech recognition supports multiple Arabic dialects, non-native accents, and Arabic-English code-switching.
  • Speech and generation: The platform provides Arabic and English text-to-speech and image generation fine-tuned to reflect Arab and Islamic preferences.
  • Retrieval services: Four RAG systems support factual attribution, post-training-cutoff information, Islamic prompts, and biographies of well-known people.

4. Pre-training Data

Fanar’s pre-training corpus combines Arabic, English, and code from diverse sources, with Arabic-specific cleaning and filtering designed to address limited high-quality Arabic data. The pipeline includes multilingual normalization, heuristic and model-based quality filtering, deduplication, and machine-translation resources for broader Arabic coverage.

  • Arabic-specific preparation: Arabic data preparation is more nuanced than English filtering because high-quality Arabic pre-training data is scarce and Arabic language characteristics require specialized handling.The pipeline adapts quality signals for right-to-left punctuation, Arabic and Hindi digits, diacritics, ligatures, Farsi characters, and decorated characters.
  • Data composition: Fanar’s pre-training data contains nearly 1 trillion tokens spanning Arabic, English, and computer code from web, scientific, encyclopedic, literary, news, and programming sources.The corpus is organized across diverse origins, with detailed source distributions presented for each domain.
  • Data composition: The corpus includes approximately 513 billion English tokens, 410 billion Arabic tokens, and 102 billion code tokens.The code subset represents around 10% of the pre-training dataset and includes common programming languages, Markdown, and GitHub Issues.
  • Quality filtering: Fanar applies 20 RedPajama quality signals, manually calibrating score cutoffs through histogram buckets and random sample inspection rather than ad-hoc thresholds.The signals measure properties including sentence and word counts and symbol-to-word ratios; samples from each bucket inform dataset-wide cutoffs.
  • Quality filtering: Arabic web filtering adjusts uniform C4 rules because a high-quality article can fail the requirement for three sufficiently long paragraphs.The example retains only one paragraph meeting the stated criterion, motivating empirically adjusted Arabic filtering rules.
  • Additional filtering and augmentation: Additional preparation removes offensive, profane, and adult content, uses model-based filters for material such as advertisements and SEO text, and selects translation systems across specialized domains.The translation systems selected were shaheen, nllb, and S-T5 after domain-specific evaluation.

5. Tokenization

The paper argues that standard tokenizers inadequately represent Arabic morphology and introduces MorphBPE, which combines morphological segmentation with BPE-style statistical merging. Evaluations report stronger morphological alignment, lower loss, and faster convergence than vanilla BPE.

  • Challenges: Arabic’s root-and-pattern morphology causes traditional BPE to create morphologically meaningless segments and unnecessary ambiguity.Such segmentation can force models to disambiguate unrelated components, complicating meaningful embedding learning.
  • MorphBPE: MorphBPE modifies BPE to respect morpheme boundaries while retaining frequency-based token merging.It first identifies structural morphemes, then selects frequent byte-pair merges that do not cross those boundaries.
  • Evaluation: The proposed Morphological Alignment Score measures how closely tokenizer outputs correspond to underlying morphological segmentation.It uses pairwise dynamic-programming alignment while preserving the order of matched tokens and morphemes.
  • Training: The Fanar tokenizer uses a vocabulary size aligned with hardware block sizes to improve token batch-processing efficiency.The vocabulary size is chosen as a multiple of 1024 for GPU and TPU processing.
  • Evaluation: Morphological BPE achieves lower training loss, highest morphological alignment, reasonable fertility, and faster convergence than vanilla BPE.The evaluation compares fertility, morphological distance, and perplexity across tokenizer variants and model sizes.

6. Modeling and Pre-training

Fanar Star and Fanar Prime use decoder-only Transformer variants trained with staged curricula and empirically selected data strategies. Ablations motivate Fanar’s filtering and dynamic language-mixing choices, while multi-stage training improves observed downstream performance.

  • Modeling: Fanar Star and Fanar Prime are 7.1B- and 8.78B-parameter decoder-only Transformers, trained from scratch and continually from Gemma-2-9B, respectively.Their architectures draw on OLMo and Llama for Fanar Star and Gemma-2-9B for Fanar Prime.
  • Ablations: Ablations train 1B-parameter models on 50–100 billion tokens and evaluate filtering and mixture choices across Arabic-translated benchmarks.HellaSwag is reported as a representative benchmark because results were consistent across HellaSwag, Open-BookQA, PIQA, and BoolQ.
  • Data filtering: About four points separate Fanar-filtered from Jais-filtered models on Arabic HellaSwag, with Fanar filtering sustaining upward performance.The recipe combines multi-stage filtering, including perplexity-based filtering and quality classification.
  • Data mixture: The 70:20 Arabic-to-English ratio improves Arabic HellaSwag by two points but degrades English benchmark performance by 6 points.Fanar therefore increases the Arabic ratio progressively toward the end of training rather than fixing one mixture throughout.
  • Training recipe: Fanar Star uses a two-stage curriculum combining multi-epoch training with a high-quality-data cool-down phase.The reported recipe totals approximately 3 trillion tokens, and loss, perplexity, and Arabic MMLU performance improve across training phases.
  • Training recipe: Fanar Prime uses a similar two-stage curriculum with 45% Arabic, 45% English, and 10% code, beginning from a competent multilingual base model.Its continual pre-training uses a single epoch before the cool-down stage, unlike Fanar Star’s scratch-training schedule.

7. Post-Training

Fanar’s post-training combines staged supervised fine-tuning, filtered bilingual data, culturally aligned synthetic data, and preference optimization. The workflow addresses quality, language balance, and Arabic cultural alignment, while user feedback identifies remaining factual and behavioral issues.

  • Objectives and strategy: Fanar’s post-training targets instruction following, meaningful dialogue, Arabic proficiency, helpfulness, harmlessness, and cultural alignment.Its strategy includes sample-level validation, multi-stage training, and value-aligned synthetic data generation.
  • Supervised fine-tuning: Public SFT data undergoes quality, translation-coherence, and value-relevance filtering before inclusion in the bilingual dataset.The process produced approximately 2.5 million instructions and dialogues across 11 categories in both languages.
  • Synthetic data: Translated public data can import English-language names, settings, traditions, and social norms into Arabic responses.Fanar addresses this mismatch by generating Arabic and English synthetic single-turn and multi-turn dialogues with local cultural contextualization.
  • Preference learning: Preference optimization improved automated benchmarks by 1–2% when preference data was processed in batches rather than in one run.Balancing Arabic and English samples and rejecting mismatched-language responses corrected occasional failures to answer in the user’s language.
  • Alignment and safety: The annealing stage provides a rapid response mechanism for harmful or culturally misaligned behaviors while preserving other capabilities through mixed training data.The authors report a slight decrease in automated benchmark performance after this stage.
  • User feedback: User feedback exposed capability, cultural, grammatical, presentation, and factual-accuracy issues that informed filtering and model-judgment refinements.The scratch-trained model had an approximately 13% dislike rate, with most dislikes related to factual accuracy.

8. Evaluation

Fanar Star and Fanar Prime are evaluated against Arabic-aware peers using standard, culturally aware, dialectal, conversational, and human assessments. Fanar Prime performs especially strongly across Arabic benchmarks and remains competitive in English.

  • Evaluation scope: Fanar Star and Fanar Prime are evaluated against Arabic-aware peer models on standard and culturally aware benchmarks.
  • Evaluation scope: Arabic coverage includes standard-language, cultural, dialectal, instruction-following, and conversational tasks, with new datasets targeting cultural and dialectal nuances.
  • Evaluation scope: The evaluation combines automatic benchmarks, open-ended and conversational assessments, and human evaluations.
  • Results: Fanar Prime achieves the best result in most Arabic benchmarks, including cultural benchmarks, with dialectal PIQA as the stated exception.
  • Results: In English, Fanar Prime beats Gemma-2-9b-it on most benchmarks except PIQA and ARC Challenge and achieves the best or second-best score overall.
  • Results: Fanar Prime Instruct ranks among the top two performers in nearly all Arabic evaluations, with the Gemma-based model particularly excelling.

9. Integrating Multimodal Support

Fanar integrates bilingual, multi-dialect speech interaction and culturally aligned image generation. Its speech systems support recognition and natural-sounding synthesis, while image fine-tuning addresses underrepresented regional concepts.

  • Platform capabilities: Fanar supports speech input, audio responses, bilingual Arabic-English processing, multiple Arabic dialects, and culturally aligned image generation.
  • Speech recognition: The Arabic-English ASR system handles dialect variation, accented speech, environmental context, and code-switching between Arabic and English or dialects.
  • Speech recognition: The speech model is trained on approximately 15,000 hours of English and Arabic data spanning MSA, dialects, broadcasts, podcasts, meetings, and academic conferences.
  • Speech recognition: Fanar ASR consistently outperforms Google ASR and Whisper-Largev2 on LAraBench and SADA datasets and matches or exceeds many reported in-domain systems.
  • Speech recognition: Fanar ASR achieves WERs of 7.5% on Libri-Clean and 12% on Libri-Other while using only 25% of the model’s vocabulary tokens.
  • Speech synthesis: The TTS system uses a non-autoregressive flow-matching diffusion-transformer architecture with in-context learning for natural, culturally adaptive, intelligible speech.
  • Image generation: Fine-tuning Stable Cascade produces more culturally nuanced images than the original model for regionally specific prompts under identical initializations.

10. Retrieval Augmented Generation

Fanar uses specialized RAG services to improve domain-specific and time-sensitive responses. Islamic RAG retrieves and reranks curated religious texts, while Recency and Biography RAGs search current or biographical information and return sourced summaries.

  • RAG services: Fanar uses RAG for Islamic content, biographical information, recent information, and attribution of fact-related prompts.
  • Islamic RAG: Islamic RAG combines a curated Islamic-literature vector database with a prompt engine for retrieval and response construction.
  • Islamic RAG: Islamic RAG chunks documents into segments of up to 2048 tokens with 50-token overlap while preserving semantic boundaries.
  • Islamic RAG: For Islamic prompts, similarity search retrieves up to twenty documents above a calibrated threshold, then reranking selects the top four relevant documents for the model context.
  • Recency and Biography RAG: An orchestrator routes recent-information and biographical prompts to dedicated RAG systems, where Fanar Prime rewrites queries, summarizes search results, and includes sources.
  • Attribution: Attribution is an opt-in, independent post-generation RAG service that fact-checks and revises Fanar responses without modifying the underlying models.

11. Discussion and Future Plans

Fanar’s first release combines two concurrently operating Arabic-centric LLMs with multimodal services, RAG systems, orchestration, and safety filtering. The authors identify data scarcity, factuality, non-agentic coordination, non-intrinsic multimodality, limited applications, and publisher-data concerns as future challenges.

  • Current platform: Fanar Star and Fanar Prime operate concurrently, with Star trained from scratch and Prime continually trained on Gemma-2 9B using the same nearly 1 trillion-token set.Star is a 7B model, while Prime is a 9B model; both address different prompt types.
  • Current platform: The platform combines speech and image generation, four specialized RAG systems, an orchestrator, and prompt- and response-level safety filters.The RAG systems address Islamic content, recent information, factual attribution, and selected biographies.
  • Challenges: Clean, broad Arabic data covering both Modern Standard Arabic and dialects remains the biggest bottleneck for scaling Arabic-centric LLMs.The authors describe coupling Fanar Star with Fanar Prime as a pragmatic response to limited Arabic coverage and English-data availability.
  • Challenges: Human testers gave Fanar a dislike rate of around 15%, with most dissatisfaction attributed to factuality-related errors.The authors report using RAG and attribution to reduce these errors, while calling for approaches that increase confidence in critical tasks.
  • Future plans: The current orchestrator is reactive rather than agentic, so future versions are planned to add reasoning, verification, and tool-calling capabilities.The authors also identify enhanced test-time computation as an emerging direction for hard tasks such as mathematical reasoning.
  • Future plans: Fanar’s multimodal capabilities are not intrinsic to one autoregressive model, and the first release includes only a small set of applications.Future plans include a unified speech, image, and text model, video generation, and broader enterprise integration driven by real use cases.
  • Future plans: Web publishers face reduced traffic and advertising revenue from LLM-driven tools, alongside emerging legal questions about compensation for copyrighted training content.The authors frame access to training data and fair compensation as an ongoing issue for generative AI development.

A. Contributions

The project spans data, pre-training, benchmarking, and other work packages, with contributions assigned to named author groups. The section also includes figures on Arabic-speaking regions and the relationship between language populations and web-data availability.

  • Contributions: The Fanar project was divided into work packages and sub-projects, with author contributions listed by subgroup.The supplied contribution passage states that names within each subgroup are ordered alphabetically.
  • Contributions: Data collection and quality were led by Hamdy Mubarak, while Fanar Star pre-training was led by Sabri Boughorbel.The contribution list separately identifies Fahim Dalvi as lead for Fanar Prime pre-training and Husrev Taha Sencar as lead for post-training.
  • Contributions: Figure 17 presents an overview of Arabic-speaking countries and compares population with web-content statistics for the world’s ten most popular languages.The figure contains a regional overview and a population-versus-web-content comparison.

C.1. Almieyar: capability-based benchmarking

Al-Mieyar introduces capability-focused Arabic benchmarking to separate linguistic strengths and weaknesses that mixed-skill evaluations can obscure. Its language segment targets Arabic linguistic understanding across five layers and dialectal variation.

  • Motivation: Al-Mieyar targets distinct LLM capabilities in Arabic language and culture, addressing the difficulty of interpreting benchmarks that entangle linguistic, reasoning, and domain skills.The broader benchmark family includes a recipe intended for adaptation to other languages and cultures.
  • Almieyar-Language: Almieyar-Language evaluates Arabic linguistic understanding because few existing benchmarks focus exclusively on language, and Holmes is English-only without detailed taxonomies or phonology coverage.The benchmark provides a structured approach to dataset generation and evaluation for Arabic and other languages.
  • Almieyar-Language: The Arabic taxonomy covers phonology, morphology, syntax, semantics, and pragmatics, with attention to characteristics specific to Arabic.These five layers are presented as core components of language knowledge and are summarized in Figure 18.
  • Almieyar-Language: Contributors speaking 16 dialects helped the benchmark emphasize lexical semantics and pragmatics, including dialect-specific elements.Fanar was evaluated on approximately one thousand high-quality questions spanning the five linguistic layers.
  • Almieyar-Overall: Almieyar-Overall contains manually reviewed Arabic questions on knowledge, reasoning, alignment, and robustness for quick feedback after supervised fine-tuning.Its hierarchical structure is illustrated in Figure 19.

C.2. Arab Cultural MCQ

The Arab Cultural MCQ dataset addresses cultural issues, values, and nuances in Arab countries through automatically generated and extensively reviewed questions. The resulting collection contains 1K questions selected from an initial 7.5K-question corpus.

  • Dataset: The dataset contains 1K multiple-choice questions on cultural issues, values, and nuances in Arab countries, curated from 7.5K questions generated by GPT-4o.Generation used relevant web pages containing content about Arabic culture.
  • Validation: The questions underwent expert review for language and factuality, followed by selection for cultural relevance, difficulty, and meaningful answer choices.The validation process included three stages of quality control.
  • Validation: Semantic clustering with bge-multilingual-gemma2 identified similar questions for manual review, producing a nonredundant 1K-question set.The highest-similarity questions were manually reviewed again to remove redundancy.
Loading 2501.13944v1…