PaperlayerView exact revision on arXiv

Source-linked AI summary

Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, Eric Xing

arXiv:2308.16149v2cs.CLcs.AIcs.LG

TL;DR

Arabic has been underrepresented in large language models, motivating models that provide strong Arabic capabilities without sacrificing English performance. The paper develops and instruction-tunes bilingual 13B Jais models using focused Arabic–English data, evaluates them across broad capabilities, and releases Jais and Jais-chat openly. Jais is reported to outperform known open Arabic models and remain comparable to state-of-the-art open English models trained on larger datasets.

  • Problem

    Arabic remains underrepresented in large language models, while multilingual models often provide limited Arabic generalization because Arabic data is dwarfed by other languages.

    Source: §1 · p. 3, §7 · p. 24

  • Method

    The paper develops a GPT-3-based 13B Arabic–English Jais model, instruction-tunes Jais-chat on Arabic and English instruction-response pairs, and adds safety-oriented tuning.

    Source: §1 · p. 3, §4.2 · p. 14, §6.1 · p. 21

  • Results

    Jais and Jais-chat outperform known open-source Arabic models and are comparable to state-of-the-art open-source English models trained on larger datasets.

    Source: §1 · p. 3, §8 · p. 25

  • Takeaways & Limitations

    The openly released Jais and Jais-chat models provide a foundation for research and development on Arabic and multilingual language applications.

    Source: §1 · p. 3, §8 · p. 25

    The models may still exhibit bias and may not produce appropriate responses to queries in languages other than Arabic and English.

    Source: §9.3 · p. 26

Abstract

from arXiv · show

We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code in various programming languages. With 13 billion parameters, they demonstrate better knowledge and reasoning capabilities in Arabic than any existing open Arabic and multilingual models by a sizable margin, based on extensive evaluation. Moreover, the models are competitive in English compared to English-centric open models of similar size, despite being trained on much less English data. We provide a detailed description of the training, the tuning, the safety alignment, and the evaluation of the models. We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs. Available at https://huggingface.co/inception-mbzuai/jais-13b-chat

1 Introduction

Jais addresses the underrepresentation of Arabic in large language models with a 13B Arabic-centric bilingual model and its instruction-tuned dialogue variant. Its focused Arabic–English training strategy supports strong Arabic performance while retaining competitive English capabilities.

Training Data

  • The authors address limited high-quality Arabic data by combining 72B Arabic tokens with 232B English tokens and code in a 395B-token pretraining corpus.Arabic is repeated 1.6 times, yielding an effective 116B Arabic-token exposure, alongside specialized Arabic filtering and cleaning.Source: §1 · p. 3

Model Design

  • Arabic constitutes 33% of pretraining, while languages beyond Arabic and English are excluded in significant proportions to prioritize Arabic fluency, cultural awareness, and English-transferred reasoning.The authors contrast this focused mixture with massively multilingual models containing more than 50 languages.Source: §1 · p. 3
  • The GPT-3-based architecture incorporates ALiBi positional encodings, SwiGLU activations, maximal update parametrization, and a tokenizer designed to weigh Arabic and English equally.These choices target longer-context extrapolation, performance improvement, smaller-model hyperparameter optimization, and balanced language treatment.Source: §1 · p. 3

Instruction Tuning

  • Jais-chat is instruction-tuned for dialogue using over 3.6M Arabic and 6M English instruction-response pairs, with additional safety-oriented tuning and deployment guardrails.The deployed interface adds safety prompts, keyword filtering, and external classifiers.Source: §1 · p. 3

Results

  • Jais is reported as superior in Arabic to similarly sized models and competitive in English despite substantially less English training data.The models are evaluated across reasoning, knowledge, misinformation, and bias benchmarks.Source: §1 · p. 3

Release and Safety

  • The release includes the base pretrained 13B Jais foundation model and the instruction-tuned Jais-chat variant, while cautioning that the models are not foolproof and require additional safety testing.The authors release these models to support further Arabic LLM research and practical development.Source: §1 · p. 3

2 Pretraining Data

Jais addresses limited high-quality Arabic data by combining a large curated Arabic corpus with English and code, using a 1:2:0.4 Arabic:English:code mix and specialized preprocessing.

2 Pretraining Data

  • The Arabic corpus combines web, Wikipedia, news, books, social content, and translated high-quality English sources.Source: §2 · p. 5
  • Arabic data is augmented with 18B translated tokens and upsampled from 72B to an effective 116B tokens.Source: §2 · p. 6
  • The English and code mixture draws from The Pile and sources including GitHub, with code included to support code-related performance.Source: §2 · pp. 6–7, Table 2 · p. 7
  • Jais pretrains on 395B tokens across Arabic, English, and programming code, with 72B Arabic tokens after adding 18B translated tokens and upsampling Arabic data 1.6 times.Source: §2 · p. 7, Table 3 · p. 8

2.1 Preprocessing Pipeline

  • Arabic preprocessing standardizes, filters, cleans, normalizes, and deduplicates documents, while using less aggressive filtering because Arabic data is scarce.Source: §2.1 · pp. 7–9

2.2 Mixing Arabic and English Data

  • Experiments identify a 1:2 Arabic–English ratio as better than Arabic-only training, while larger models benefit more from English data.Source: §2.2 · p. 9

3 Model

Jais uses a GPT-style decoder-only transformer with Arabic-aware tokenization and several efficiency-oriented design choices, then selects model and data configurations through smaller-scale experiments.

3.1 Model Architecture

  • Jais uses a causal decoder-only transformer with ALiBi, SwiGLU, maximal update parametrization, and a custom bilingual tokenizer.Source: §3.1 · pp. 9–10
  • The Jais tokenizer is trained with balanced Arabic and English data to produce meaningful subwords and reduce Arabic over-segmentation.Source: §3.1 · p. 10

3.2 Model and Training Hyperparameters

  • Hyperparameters are tuned on a roughly 40M-parameter model and transferred to the 13B model using maximal update parametrization.Source: §3.1 · p. 10, §3.2 · p. 11
  • Training packs sequences to 2,048 tokens, concatenating shorter documents and separating them with <|endoftext|>.Source: §3.2 · p. 10

3.3 Learnings and Observations

  • The 13B model with a 1:2 Arabic–English mix outperforms the 30B Arabic-only model on Arabic cross-entropy loss, while English can hurt smaller models.Source: §3.3 · p. 11, Figure 3 · p. 11

3.4 Training Infrastructure

  • All training, tuning, and instruction-tuning experiments run on the Condor Galaxy 1 supercomputer using Cerebras wafer-scale systems.Source: §3.4 · p. 12

4 Instruction-Tuning

Jais-chat adapts bilingual instruction–response data for dialogue while applying Arabic-specific data construction and answer-focused training. The setup combines translated and native Arabic resources with English datasets, safety examples, and distinct templates for single-turn and multi-turn interactions.

4 Instruction-Tuning

  • Instruction tuning adapts the language model for dialogue-style interaction because pretrained models do not inherently interpret user instructions or sustain such exchanges.The resulting instruction-tuned model is called Jais-chat.Source: §4 · p. 12

4.1 Instruction-Tuning Data

  • 10M prompt–response pairs combine 4M Arabic and 6M English examples across single-turn and multi-turn formats.The datasets cover diverse domains and include English and Arabic instruction-tuning resources.Source: §4.1 · p. 12

4.1.1 English Instruction-tuning Datasets

  • English instruction tuning draws on task, dialogue, question-answering, safety, mathematics, writing, and programming datasets.Sources include Super-NaturalInstructions, P3, xP3, Natural Questions, dialogue data, and specialized task collections.Source: §4.1.1 · pp. 12–14
  • The English dataset inventory includes curated or filtered assistant data, harmful-content refusal examples, mathematical tasks, essay and poem writing, and code-oriented questions.The collection also includes regional UAE-focused NativeQA and SafetyQA resources.Source: §4.1.1 · pp. 13–14

4.1.2 Arabic Instruction-Tuning Datasets

  • Arabic instruction tuning combines translated English datasets with native Arabic examples for regional knowledge, named-entity recognition, and safety.The authors manually exclude translated tasks involving word counting or primarily translation because their properties may not survive translation.Source: §4.1.2 · p. 14

4.2 Instruction-Tuning Setup

  • Each training instance wraps a prompt and response in special markers, uses separate single-turn and dialogue templates, and masks prompt loss so learning targets answer tokens.Examples are padded rather than packed, while retaining the autoregressive pretraining objective.Source: §4.2 · p. 14, Figure 4 · p. 15
  • The instruction-tuning templates distinguish human input from expected responses using additional special markers.This formatting supports the model’s need to separate prompts from answers during training.Source: §4.2 · p. 14

5 Evaluation

The evaluation spans Arabic and English downstream capabilities and Arabic open-ended generation. Jais-chat leads Arabic benchmark comparisons, remains competitive in English despite less English data, and approaches larger models on several Arabic generation categories while retaining weaknesses on harder reasoning tasks.

5.1 Downstream Evaluation

  • The downstream evaluation uses zero-shot accuracy across knowledge, commonsense reasoning, misinformation, and bias, with Arabic additions including EXAMS and LiteratureQA.Arabic data combine machine translation, native-speaker translation of MMLU, and manually constructed Arabic questions.Source: §5.1 · pp. 15–16, Table 8 · p. 16
  • +11.7 to +15.3 absolute points over monolingual Arabic models make Jais-chat (13B) the strongest Arabic model across the reported criteria.The evaluation covers knowledge, commonsense reasoning, misinformation, and bias in zero-shot settings.Source: §5.1 · pp. 15–16, Table 9 · p. 17
  • +1.9 points separates Jais-chat (13B) from Jais-chat (6.7B), after the smaller model already exceeds baselines by +3.5 to +10.9 points.BLOOMz (7.1B) is the strongest baseline at 42.9 average accuracy, ahead of mT0-xxl (13B) at 40.9.Source: §5.1 · p. 16
  • +0.7, +3.2, and +1.9 are the downstream gains from instruction tuning for Jais-chat at 1.3B, 6.7B, and 13B, respectively.Instruction tuning improves corresponding base models except Falcon (7B).Source: §5.1 · p. 16
  • 57.3 vs. 57.7 places Jais-chat (13B) on par with LLaMA2-chat (13B) in English despite 232B versus 2T English word tokens.Jais-chat also exceeds other baselines by +2.6 to +7.2 points and improves over its base model with instruction tuning.Source: §5.1 · p. 17, Table 10 · p. 18

5.2 Generation Evaluation

  • BLOOM, AraT5, AraT5-v2, and AraBART struggle with meaningful Arabic instruction following, while BLOOMz and LLaMA2 average below 1 against Jais-chat (13B).This contrasts with BLOOMz’s competitiveness on downstream Arabic task evaluation.Source: §5.2 · p. 19
  • Jais-chat (13B) trails ChatGPT and Claude overall by around 400 points, despite those models being 4–13 times larger.The comparison aggregates pairwise GPT-4 scores across 80 Arabic open-ended questions, with a maximum total of 4,000.Source: §5.2 · p. 19, Figure 5 · p. 19
  • 35 and 114 scores separate Jais-chat from Claude and ChatGPT, respectively, on common-sense, knowledge, writing, and generic questions out of 2,000.Jais-chat is especially close on writing, while it performs worse on counterfactual, Fermi, roleplay, and math-and-coding tasks.Source: §5.2 · pp. 19–20, Figure 6 · p. 19

6 Safety

Jais-chat’s safety design combines instruction-tuning, system prompts, external classifiers, and keyword filtering. These measures target harmful, discriminatory, privacy-sensitive, misleading, illegal, and otherwise unsafe interactions, while the authors acknowledge deployment limitations.

6 Safety

  • Safety measures are applied at multiple development stages, including model instruction tuning and practical safeguards for interactive applications.The application-oriented safeguards are presented as examples for developers.Source: §6 · p. 20

6.1 Safety via Instruction-Tuning

  • Safety instruction tuning teaches Jais-chat to reject discrimination, privacy leakage, harmful advice, malicious use, emotional manipulation, and selected sensitive-topic discussions.The principles are implemented through examples designed to produce helpful but harmless responses.Source: §6.1 · p. 21

6.2 Safety via Prompting

  • The user-facing interface adds Arabic and English system prompts that prohibit harmful, illegal, toxic, misleading, explicit, and privacy-violating content.The prompts also instruct the model to acknowledge uncertainty and avoid professional advice.Source: §6.2 · pp. 21–22

6.3 Safety via External Models

  • External hate and offensive-speech classifiers intercept inputs before Jais-chat when detected content is hateful or offensive.Validation performance is 94.8% accuracy and 91.04% F1 for offensive speech, versus 96.6% accuracy and 81.02% F1 for hate speech.Source: §6.3 · p. 22

6.4 Safety via Keywords

  • Keyword filtering uses curated objectionable-word lists and regex patterns to return safe refusals without calling Jais-chat.The mechanism is intended for real-time detection of obscene or explicit content.Source: §6.4 · p. 23
  • Keyword filtering can produce false positives and must adapt to the dynamic nature of language usage.The authors describe continuous keyword-list refinement and exploration of more advanced NLP techniques.Source: §6.4 · p. 23

7 Related Work

Related work positions Jais within Arabic language modeling, autoregressive LLMs, instruction tuning, and multilingual evaluation. The paper emphasizes that existing Arabic benchmarks and public large models leave gaps in knowledge, commonsense, scale, and zero-shot assessment.

Arabic Language Models

  • Arabic encoder-only, decoder-only, and encoder–decoder models have demonstrated competitiveness after fine-tuning on natural-language understanding and generation tasks.The cited families include AraBERT, ARAGPT2, AraT5, and AraBART.Source: §7 · p. 24
  • Before Jais, no public Arabic language model reportedly exceeded one billion parameters while demonstrating broad generalization and robust zero-shot capabilities.Multilingual models often underrepresent Arabic and consequently show weaker Arabic generalization than dedicated models.Source: §7 · p. 24

Large Language Models

  • Recent large language models remain English-centric despite multilingual training, with GLM identified as an exception designed for Chinese and English.Jais is presented as an open-source Arabic-centric alternative.Source: §7 · p. 24
  • Jais and Jais-chat follow the autoregressive, decoder-only paradigm used by GPT-series, LLaMA, BLOOM, and Falcon models.The paper situates this design among autoregressive, autoencoding, and encoder–decoder pretraining frameworks.Source: §7 · p. 24

Instruction-Tuning

  • Instruction tuning improves cross-task generalization, while RLHF-based combinations target factuality and reduced toxicity through reward optimization.Prior open models such as BLOOMz and LLaMA2 exemplify instruction–response fine-tuning approaches.Source: §7 · p. 24
  • Instruction prompts may be human-designed, autonomously generated, refined through follow-up instructions, or structured as chain-of-thought prompts.These prompt origins and refinements represent the broader instruction-tuning design space surrounding Jais-chat.Source: §7 · p. 24

Evaluating Large Language Models

  • LLM evaluation commonly combines exam-style knowledge tests, commonsense datasets, and programming benchmarks, while whole-text quality remains important.The paper motivates GPT-4 pairwise assessment as an alternative to resource-intensive or variable-quality crowd evaluation.Source: §7 · p. 25
  • Existing Arabic benchmarks emphasize natural-language understanding but omit knowledge and commonsense evaluation, complicating assessment of Jais.The paper addresses this gap by adapting English-style datasets through translation and Arabic dataset construction.Source: §7 · p. 25

8 Conclusion

Jais and Jais-chat are Arabic-English bilingual language models introduced to expand Arabic NLP capabilities. The authors report strong Arabic performance and competitiveness with English open models, while encouraging research and development using the released models.

  • Jais and Jais-chat are Arabic-English bilingual large language models presented as new open models for Arabic NLP.Source: §8 · p. 25
  • The models support generative and downstream tasks in both Arabic and English, including reasoning, sentiment analysis, irony detection, and hate speech detection.Source: §8 · p. 25
  • The authors report that the models outperform known open-source Arabic models and are comparable to state-of-the-art open-source English models trained on larger datasets.Source: §8 · p. 25
  • The paper encourages researchers, hobbyists, and enterprise developers to experiment with Jais, particularly for multilingual and non-English applications.Source: §8 · p. 25

9 Release Notes

The release notes describe Jais as an openly available Arabic-English model intended for research, development, and commercial applications. They also define out-of-scope uses involving malicious content, unsupported languages, high-stakes decisions, bias, and sensitive information.

9 Release Notes

  • Jais is released under the Apache 2.0 license for researchers, hobbyists, and enterprise developers, subject to applicable policies, laws, and regulations.Source: §9 · p. 25

9.1 Intended Use

  • The model is intended for Arabic NLP research, Arabic-focused businesses, and developers integrating Arabic capabilities into applications.Source: §9.1 · p. 26
  • Potential commercial uses include chat assistants, downstream NLU/NLG tasks, customer service, and process automation.Source: §9.1 · p. 26

9.2 Out-of-Scope Use

  • Users should not use Jais to generate harmful content, spread misinformation, conduct illegal activities, or handle personal, confidential, or sensitive information.Source: §9.2 · p. 26
  • Because Jais is optimized for Arabic and English, users should not assume equal proficiency in other languages or dialects.Source: §9.2 · p. 26
  • High-stakes medical, legal, financial, or safety-critical decisions require human oversight, and the model may still exhibit biases despite mitigation efforts.Source: §9.2 · p. 26, §9.3 · p. 26

9.3 Biases, Risks, and Limitations

  • The model is intended to assist Arabic and English speakers and may produce inappropriate responses for queries in other languages.Source: §9.3 · p. 26

A Detailed Zero-Shot Evaluation Results

The appendix provides detailed zero-shot evaluation results for Arabic and English. Arabic results are reported in Table 12 and English results in Table 13, with Table 12 defining its averaging and translation conventions.

B Jais-chat Response Examples

The appendix presents Arabic and English Jais-chat response examples, including an Arabic instruction-response table with English translations. The examples cover questions about Abu Dhabi attractions and other conversational prompts.

B Jais-chat Response Examples

¯ ÉJ.k. Èñ£@ ñë AÓ ? ÕËAªË@

B Jais-chat Response Examples

  • One example asks about famous tourist areas in Abu Dhabi and receives a response listing attractions including the Sheikh Zayed Grand Mosque, the Corniche, and Louvre Abu Dhabi.Source: B Jais-chat Response Examples · p. 41

¯ ÉJ.k. úΫ@

The section presents example prompts and responses covering factual, geographic, and problem-solving questions, including English examples about mountains, Abu Dhabi events, chess, and Burj Khalifa.

ZA¿ Y Ê Ë ?ú «A J¢“B@

The section provides Arabic and English prompt-response examples across informative, creative, instructional, role-play, tricky, and problem-solving settings, alongside model cards for Jais and Jais-chat.

ZA¿ Y Ê Ë ?ú «A J¢“B@

C Model Cards

Related papers

Loading 2308.16149v2…