PaperlayerView exact revision on arXiv

Source-linked AI summary

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

Hao Yu, Tianyi Xu, Michael A. Hedderich, Wassim Hamidouche, Syed Waqas Zamir, David Ifeoluwa Adelani

arXiv:2601.06395v3cs.CL

TL;DR

Open models underperform especially on African languages, while CPT gains on reasoning remain limited by uneven low-resource data coverage. AfriqueLLM adapts five model families and scales to 20 African languages using 26B-token CPT with varied data mixtures. The central result is that data composition drives gains, while architecture and task alignment matter more than base multilingual coverage alone.

  • Problem

    Open models lag proprietary systems most on African languages, and CPT improvements on demanding reasoning tasks are limited by uneven domain and knowledge coverage.

  • Method

    AfriqueLLM applies 26B-token CPT to models spanning Llama 3.1, Gemma 3, and Qwen 3 while varying mixtures of monolingual, math, code, and synthetic data.

  • Results

    Data mixture is the strongest determinant of CPT gains; adding math, code, and synthetic translated data consistently improves performance, while architecture dominates scale across model families.

  • Takeaways & Limitations

    Task-aligned data combined with robust architectures provides a more dependable recipe for African-language adaptation than relying on base-model multilingual coverage.

  • Takeaways & Limitations

    The study covers 20 African languages, experiments only up to 14B parameters, and focuses on base-model CPT without instruction tuning.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present \texttt{AfriqueLLM}, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm) and [Github](https://github.com/McGill-NLP/AfriqueLLM).

1 Introduction

AfriqueLLM addresses persistent performance gaps for African languages by studying continued pre-training data mixtures and model architectures. The study finds that task-aligned data composition drives gains, while architecture and scale shape outcomes across models.

  • CPT gains remain limited on mathematical reasoning and knowledge-based question answering because low-resource languages have uneven domain and knowledge coverage.
  • AfriqueLLM adapts open models to 20 African languages through continued pre-training on 26B tokens across multiple architectures and scales.The study includes Llama 3.1, Gemma 3, and Qwen 3 backbones.
  • The evaluation varies mixtures containing math, code, and synthetic translated data to analyze their effects on downstream performance.
  • Adding math, code, and synthetic translated data consistently improves performance, including on reasoning-oriented evaluations.
  • Within architectures, larger models generally perform better, but across architectures scale alone is not predictive and task-aligned data is more dependable.
  • The best models preserve high-resource-language performance after CPT and perform strongly on long-context tasks such as document-level translation.

2 Related Work

Related work establishes data quality, heterogeneous mixtures, and continued pre-training as central approaches for improving multilingual and low-resource language models. AfriqueLLM extends this line by systematically studying CPT mixtures for African languages.

  • Recent multilingual data pipelines improve quality and coverage, including resources spanning over 1,000 languages and specialized African-language datasets.
  • Training recipes increasingly adjust web, code, and math ratios through multi-stage curricula and specialized mixtures.
  • Open-weight Llama 3.1, Qwen 3, and Gemma 3 provide strong foundations for adapting underrepresented languages through CPT.
  • Prior work shows CPT can produce strong African-language models and substantial gains without training from scratch.
  • AfriqueLLM builds on CPT by exploring new data mixtures for models adapted to Africa’s linguistic and cultural diversity.

3 AfriqueLLM: Data & Training Recipe

AfriqueLLM uses a 26B-token CPT corpus combining African-language text with code, mathematics, synthetic translations, and filtered parallel data. The recipe balances language coverage, task-relevant knowledge, and training efficiency.

  • Dataset Curation: The corpus combines monolingual text, code, mathematics, and domain-specific synthetic data to cover downstream knowledge and skill distributions.
  • Dataset Curation: Monolingual data combines FineWeb2, WURA, and MADLAD-400, while document-level sources increase contextual diversity and lower-resource coverage.
  • Sampling Strategy: UniMax caps high-resource languages at approximately 1B tokens and upsamples lower-resource languages for up to five epochs.
  • Code and Mathematics: Code and mathematics contribute approximately 1B tokens each from Python code and educational mathematics sources to strengthen reasoning and logical abilities.
  • Synthetic Data: Synthetic data adds 324M translated tokens from diverse web domains and mathematical reasoning questions.
  • Dataset Curation: A translation-quality threshold of 0.7 reduces an initially collected 1B bilingual pairs to a high-quality subset of 4M samples.
  • Dataset Curation: The curation process produces CMS and CMSP mixtures totaling 25.2B and 25.6B tokens, respectively.
  • Hyperparameter Tuning: Hyperparameter studies identify 5e-5 as the preferred learning rate and 16k tokens as the best tested context length for reasoning performance.

4 Evaluation Setting

The evaluation measures African-language capabilities across translation, reasoning, knowledge, classification, and comprehension using standardized multilingual benchmarks. It compares Gemma, Llama, and Qwen base models to assess data mixtures and scaling.

  • Benchmarks: AfroBench covers 64 languages across 15 tasks, while the study uses its AfroBench-Lite subset for efficient evaluation.
  • Benchmarks: AfroBench-Lite evaluates math, knowledge, natural language inference, reading comprehension, translation, intent classification, and topic classification.
  • Benchmarks: The evaluation expands coverage beyond the original 14 languages to all African languages represented in each dataset or task.
  • Metrics: Pretrained checkpoints use five-shot evaluation, except AfriMGSM, which uses the default eight-shot setting; Flores uses SSA-COMET for translation scoring.
  • Baseline Models: Baselines include Gemma 3, Llama 3.1, and Qwen 3, with Qwen selected partly for strong mathematical reasoning despite limited native African-language support.
  • Experimental Pipeline: The experimental pipeline validates mixtures on Gemma 3 4B and 12B before scaling to Llama 3.1 8B and Qwen 3 4B, 8B, and 14B.

5 Experiments Results

The experiments show that CPT outcomes depend strongly on data composition and architecture. Structured and synthetic data recover reasoning performance, while Qwen models deliver especially large gains with comparatively limited high-resource degradation.

  • 5.1 Data Mixture Ablation: Adding code and math reverses the reasoning decline caused by monolingual-only CPT, recovering performance across challenging tasks.For 12B models, configurations containing code and math recover reasoning performance; for 4B models, code and math improve all evaluated tasks.
  • 5.1 Data Mixture Ablation: CMS achieves the highest reported MGSM and Flores scores, whereas adding parallel data reduces performance at 12B scale.CMS reaches MGSM 30.9 and Flores 66.2; CMSP performs worse after adding parallel data.
  • 5.1 Data Mixture Ablation: Synthetic data and structured reasoning data improve CPT generalization, with high-quality synthetic data more effective than noisy parallel data for larger models.This quality-versus-scale pattern motivates CMS as the primary recipe.
  • 5.2 Impact of Model Selection and Scaling: Qwen 3 achieves relative gains of 74.4% for 4B, 76.5% for 8B, and 57.8% for 14B, exceeding the corresponding Gemma gains.AfriqueQwen-14B also scores 63.58 overall, surpassing Gemma 3 27B at 60.71 despite having less than half the parameters.
  • 5.2 Impact of Model Selection and Scaling: Qwen models preserve high-resource performance better than other families, with HRL degradation of −1.6% for Qwen 3 8B and −3.8% for 14B.Llama 3.1 8B declines by −10.2% on average, while Gemma models decline by −5.6% to −6.2%.
  • 5.3 Language-wise Analysis: CPT gains are strongest for covered African languages, while Qwen 3 8B exceeds +25 absolute points in 10 of 11 covered languages.The largest reported Qwen gain is +37.1 in Hausa; transfer to unseen languages is limited, while Qwen 3 8B maintains near-parity on high-resource languages at +0.2.
  • 5.4 Document-Level Translation: Afrique models outperform the document-translation SFT baseline in several directions, with AfriqueGemma-12B reaching 60.2 for eng→xx versus 47.8.For xx→eng, AfriqueQwen-14B reaches 73.5 and AfriqueGemma-12B 73.3, above the 68.0 SFT baseline.
  • 5.5 Qwen 3 Family: A more multilingual Qwen 3.5 base raises absolute post-CPT performance but reduces relative gains, while expanding Code and Math recovers math performance.Qwen 3.5 4B reaches 57.12 versus 54.94 for Qwen 3 4B, and ExtendedCM raises AfriMGSM from 30.47 to 34.17.

6 Conclusion

AfriqueLLM adapts open language models to 20 African languages through 26B-token CPT, showing that data mixtures and base-model capabilities strongly shape post-adaptation performance. The best configurations combine monolingual, code, math, and synthetic data while preserving reasoning and achieving strong multilingual results.

  • Combining monolingual text with code, math, and synthetic data yields state-of-the-art results while preserving reasoning.
  • Qwen 3’s strong capability priors outweigh multilingual coverage after CPT, despite minimal African-language support.
  • AfriqueQwen-14B scores 63.58, surpassing Gemma 3 27B at 60.71 with less than half the parameters.
  • High-quality synthetic data provides a scalable bridge for low-resource languages, with AfriqueQwen-14B surpassing the 54B NLLB-MoE on translation.

7 Limitations

The study’s scope is constrained by language coverage, model scale, training stage, hyperparameter search, and training stability. These boundaries limit how broadly its adaptation results and efficiency claims can be generalized.

  • The study covers 20 African languages, leaving hundreds unsupported, especially languages with minimal digital presence.
  • Experiments are limited to models up to 14B parameters, so larger models may exhibit different adaptation dynamics and performance.
  • The work focuses on base-model CPT without instruction tuning, while scarce high-quality African-language instruction data remains a deployment bottleneck.
  • Scaling to 12B+ models prevents exhaustive hyperparameter search, so heuristics from smaller models may not be optimal across architectures.
  • Intermittent gradient-norm spikes indicate latent optimization instabilities, although training did not diverge.
  • The framework reaches 31–34% Model FLOPs Utilization, leaving room for improvement through specialized training frameworks.

A Data Details

The data pipeline selects languages and domains for CPT, combines translated and mathematical content with general translation prompts, and uses ablations to tune training choices. These details support broad topical coverage and systematic configuration selection.

  • Languages are selected using a 90M-token threshold to ensure sufficient data for meaningful linguistic adaptation.
  • Synthetic data spans 11 domains, with translation guided by prompts described in the appendix.
  • The general translation prompt preserves meaning, tone, formatting, inline markup, numerals, and named entities while requiring fluent, complete output.
  • The mathematical-reasoning prompt translates problems, reasoning, and answers while preserving numbers, formulas, notation, markup, named entities, and tone.
  • The ablation study evaluates hyperparameters for continued pretraining on African languages.
  • A learning rate of 5e-5 is selected as optimal for Gemma 3 4B based on average scores across low-resource languages.
  • A 16k context window yields the best AfriMGSM performance under the selected learning rate.
  • Cosine-scheduler experiments vary minimum learning rate and warmup steps using a fixed 16k context size.

B.2 Training Configuration

Training uses the LLaMA-Factory framework with full CPT, 16k sequences, distributed optimization, and a cosine schedule, while the accompanying tables document data, languages, ablations, and efficiency metrics.

  • The training configuration uses LLaMA-Factory for continued pretraining of AfriqueLLM models.
  • The configuration uses Gemma 3 as a model path, empty templates, packed examples, and a 16k-token cutoff length.
  • Training performs full-parameter updates while freezing the vision tower and multimodal projector, with DeepSpeed ZeRO-1 distributed training.
  • The run logs every 10 steps, saves checkpoints every 1,000 steps, plots loss, and reports to Weights & Biases.
  • The dataset tables document language selection, token counts, synthetic-data domains, evaluation ablations, training efficiency, and task-language coverage.
  • The training run uses a 5.0e-5 learning rate, one epoch, cosine scheduling, 0.1 weight decay, and bf16 computation.
  • FlashAttention 3 and Liger kernels are enabled for the training configuration.

C.2 Newly Added Languages in the 50-Language Version

The +50LANGS configuration expands the CPT corpus from 20 to 50 African languages, retaining only newly added languages with non-zero final tokens. Evaluation covers only the subset of these languages represented in each benchmark.

  • Corpus expansion: The +50LANGS configuration expands the CPT corpus from 20 to 50 African languages.The expansion includes previously held-out languages with non-zero Final Used Tokens after UniMax upsampling.
  • Corpus expansion: Languages with zero final tokens, including Umbundu, Tumbuka, Nuer, and Chokwe, remain excluded.
  • Evaluation coverage: Only a subset of the newly added languages is covered by the AfroBench-Lite evaluation suite.Table 15 identifies the newly added languages contributing to the “30 newly added languages” evaluation block in Table 7.

D Detailed Experimental Results

The detailed-results section organizes evaluation across multilingual reasoning, understanding, inference, reading comprehension, translation, intent classification, and topic classification benchmarks. It reports benchmark-specific results using the stated few-shot settings and metrics.

  • Benchmark coverage: The detailed experiments cover multilingual grade-school math, language understanding, natural-language inference, and reading comprehension.AfriMGSM uses 8-shot evaluation, while AfriMMLU, AfriXNLI, and Belebele use 5-shot evaluation.
  • Benchmark coverage: Translation is evaluated in both English-to-African and African-to-English directions on FLORES-200.The section includes SSA COMET and chrF++ scores for both translation directions.
  • Benchmark coverage: The experiments also include 5-shot intent classification and topic classification benchmarks.These evaluations use Injongo Intent Classification and SIB-200 topic classification.
Loading 2601.06395v3…