PaperlayerView exact revision on arXiv

Source-linked AI summary

Wolof-Adapted Industry-Ready Open-Weight Language Models

Gwénolé Lecorvé, Delphine Charlet, Pierre Adam, Floriane Behanzin, Lionel Delphin-Poulat, Johannes Heinecke, Géraldine Damnati, Pelin Kaytmaz, Nourredine Khaous, Charbel Kindji, Meriam Maadi, Abdoulaye Mbaye, Claire Perroux, Ismaël Rousseau, Papa Séga Wadé, Gilles Xiberras, Olivier Zhang, Malek Zitouni

arXiv:2610.20824v1cs.CLcs.AI

TL;DR

Wolof remains weakly supported by open-weight language models, while API models raise cost and governance constraints for local deployments. The paper adapts open-weight models through Wolof instruction tuning, with preference alignment and model merging as complementary stages. Evaluations report strong Wolof capabilities and retention of broader skills, especially at 8B and 14B.

  • Problem

    Wolof remains under-served by current LLMs, while commercial API models can be costly or incompatible with local data-governance and sovereignty requirements.

    Source: §1 · p. 1

  • Method

    The paper adapts open-weight models to Wolof primarily through SFT, with preference alignment and model merging used as complementary approaches to balance Wolof and general capabilities.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Adaptation substantially raises Wolof understanding and generation scores over unadapted Qwen3 models, with 4B–14B models also preserving broader capabilities while the 0.6B model degrades on several skills.

    Source: §5.1 · pp. 5–6

  • Takeaways & Limitations

    The 8B and 14B models narrow the gap with commercial APIs, and the models and selected materials are ready for public non-commercial release.

    Source: §6 · p. 6

    Final experiments use the Qwen family, so cross-family generalization should be tested again with the full recipes.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Wolof is spoken by 23M people, mostly in Senegal, but remains weakly supported by open-weight language models, while API models raise cost, governance, and sovereignty constraints for local deployments. In this paper, we present 0.6B-14B open-weight models adapted to Wolof through instruction tuning, preference alignment, and model merging. After presenting the pipeline and data, extensive experiments on 63~public and in-house tasks show good performance on Wolof understanding and generation while remaining effective on other languages and cross-lingual skills, with the 8B and 14B models being particularly competitive with substantially larger commercial systems. The models are currently being evaluated in internal customer-care pilots and are publicly available at https://huggingface.co/collections/Orange/wolof-lm .

1 Introduction

Wolof remains underserved by language models despite its broad use in Senegal, while commercial APIs raise cost and local data-governance concerns. The paper adapts open-weight models for Wolof and evaluates them for industrial use.

1 Introduction

  • Wolof has 23M speakers and remains under-served by current LLMs, limiting applications in banking, telecommunications, and public services.Commercial API-based models offer some Wolof capability but may be too costly or incompatible with local data-governance and sovereignty requirements.Source: §1 · p. 1
  • The paper adapts open-weight small language models to Wolof and evaluates them on industrially realistic use cases.Customer-relationship automation includes offline conversation analysis, FAQ answering, transactional support, and interactive chat.Source: §1 · p. 1
  • The work provides Wolof-adapted Qwen models from 0.6B to 14B, a broad evaluation, and 222K instructions plus some in-house test sets.The models are being integrated into internal customer-care pilots, with a target customer base of several million and the goal of reducing dependence on commercial systems.Source: §1 · p. 1

2 Related Work

Prior work spans African-language model training and adaptation, but Wolof resources and systematic evaluation remain limited. This paper positions its approach around Wolof instruction tuning and a combination of public and manually checked evaluation.

2 Related Work

  • Recent African LLM surveys and benchmarks report coverage, tokenization, and evaluation gaps, while Wolof resources remain mostly task-specific.Examples include banking-domain intent classification (Kandji et al., 2026) and resource surveys (Mbaye et al., 2025).Source: §2 · p. 1
  • African-language modeling includes training from scratch, multilingual instruction tuning, continued pre-training, and delta-based merging.InkubaLM trains compact models from scratch (Tonja et al., 2024); Aya studies multilingual instruction tuning (Singh et al., 2024a); AfriqueLLM and Lugha-Llama study continued pre-training (Yu et al., 2026; Buzaaba et al., 2025); DIPLomA combines target-language continued pre-training with delta-based merging.Source: §2 · p. 2
  • Public Wolof evaluation resources are sparse but growing, and translated or English-pivoted benchmarks may not fully reflect in-language quality.The paper complements automatic evaluation with manually checked in-house tests and blind human assessment by Wolof speakers; it also motivates a controlled comparison with Oolel-v0.1, which lacks public training data or detailed evaluation.Source: §2 · p. 2

3 Adaptation Method

The adaptation pipeline uses full fine-tuning, beginning with Wolof-centered supervised instruction tuning and seeking to preserve general behavior. Preference alignment and model merging are evaluated as complementary post-SFT approaches, with the final experiments using Qwen3 models.

3 Adaptation Method

  • SFT acquires Wolof capabilities using a mixed corpus intended to preserve general behavior; preference alignment and merging are tested as complementary ways to balance those capabilities.The contribution of these extra stages depends on model capacity, with larger positive effects as capacity grows; the pipeline uses full fine-tuning rather than LoRA.Source: §3 · p. 2

3.1 Instruction Tuning

  • Native-speaker assessment of 50 Google Translate outputs scored meaning fidelity 4.59/5 and Wolof quality 4.78/5.Translated data is treated as scalable but imperfect supervision, counterbalanced with native and bilingual sources and Wolof-specific evaluation.Source: §3.1 · p. 2
  • The 446K-sample, 109.4M-token instruction corpus combines native and bilingual Wolof resources with translated public instructions and four skill-guided blocks.Aya's Wolof instructions were excluded from training and reserved for evaluation to avoid contaminating a public Wolof benchmark.Source: §3.1 · pp. 2–3
  • A continued-pretraining stage was investigated but excluded from the main recipe because the collected raw corpus of 13M Wolof tokens was insufficient.Tokenization adaptation was also explored, but was not included in the main recipe.Source: §3.1 · p. 3

3.2 Preference Alignment and Model Merging

  • DPO uses 233,804 preference pairs, 98% in English and none in Wolof, to reinforce general assistant behavior after SFT.Model merging uses linear interpolation to limit movement from the original model and reduce the risk of catastrophic forgetting; coefficient sweeps select 0.4 base and 0.6 adapted weights after SFT, or 1/3 coefficients when mixing base, post-SFT, and post-DPO models.Source: §3.2 · p. 3

3.3 Training Setup

  • Final experiments use Qwen3 models from 0.6B to 14B after Qwen was selected as the best tradeoff between performance and stability among tested families.SFT and DPO used internal GPU nodes and distinct training configurations; checkpoints were selected by validation loss.Source: §3.3 · p. 3

4 Benchmarking Tasks and Skills

The evaluation suite targets Wolof understanding and generation while checking English and French retention and broader assistant skills. It combines public benchmarks with in-house customer-service tasks and uses automatic metrics alongside human-checked judgment of freeform outputs.

4 Benchmarking Tasks and Skills

  • The 63-task suite combines public academic benchmarks with in-house customer-service tasks; 35 tasks include Wolof and 13 use in-house data.The in-house tasks cover FAQ retrieval, intent routing, privacy-sensitive entity extraction, summarization, and response generation.Source: §4 · pp. 4–5
  • The evaluation suite groups tasks by family and their contribution to target skills, alongside task inputs, outputs, sample counts, and metrics.The automatic-results table reports normalized scores and ranks by language and skill, with higher scores and lower ranks indicating better performance.Source: Table 3 · p. 5
  • Wolof understanding and generation are the primary targets, while English and French monitor multilingual behavior and other skills cover assistant interaction and regression checks.Instruction following, knowledge, chat, formatting, and safety reflect user interaction needs; function calling and reasoning target agentic AI, while code and long context are broader checks.Source: §4 · p. 5
  • Deterministic metrics score reliably matchable outputs, while LLM judges assess freeform generation and are treated as comparative signals rather than standalone evidence.Native speakers accepted 95% of Gemini-2.5-flash scores on 450 Wolof responses; evaluations use fixed-seed prompt templates and temperature-zero decoding.Source: §4 · p. 5

5 Results

Wolof adaptation raises understanding and generation from near-random unadapted baselines into the range of commercial APIs. At 4B–14B, those gains generally coexist with broader capability retention, while human ratings place the 8B model close to Gemini.

5 Results

  • The section reports each size’s best final released setup; Table 3 compares adapted and unadapted Qwen3, APIs, and prior Wolof-aware models using per-task normalized scores.Source: §5 · p. 5, §5.1 · p. 5

5.1 Automatic evaluation

  • Adapted Qwen3 models significantly raise Wolof understanding and generation over near-random unadapted baselines, reaching the range of commercial APIs.Source: §5.1 · pp. 5–6
  • At 4B–14B, Wolof gains coexist with preservation of English, French, safety, reasoning, long-context, and coding skills; function calling declines mainly through excess tool calls.Source: §5.1 · p. 6
  • Wolof improvements span classification, labelling, and generation, while SFT is the main adaptation driver.Source: §5.1 · p. 6
  • DPO and merging effects vary with model size; hyperparameters tuned only at 8B may need size-specific settings.Source: §5.1 · p. 6

5.2 Human Evaluation via Arena

  • Human ratings slightly favored Gemini, but were very close to the final 8B model; Oolel lagged slightly and the unadapted model clearly lagged.For pure Wolof quality, the final model was described as clearly comparable to closed models.Source: Table 4 · p. 6, §5.2 · p. 6

6 Conclusion and Further Improvements

The paper introduces open-weight Wolof-adapted models from 0.6B to 14B parameters, with the 8B and 14B models narrowing the gap with commercial APIs. It reports practical insights into adaptation-stage effects and identifies function calling as a next step for agentic capabilities.

6 Conclusion and Further Improvements

  • The 0.6B–14B open-weight models show strong Wolof capabilities while retaining multilingual and operational abilities, with 8B and 14B models narrowing the gap with commercial APIs.Source: §6 · p. 6
  • The models, training datasets, and selected in-house test sets are ready for public release for free non-commercial use.Source: §6 · p. 6
  • The results offer practical insights into model size, SFT, preference alignment, and merging; agentic work has begun with GRPO and verifiable rewards, especially targeting function calling.Source: §6 · p. 6

Limitations

The paper identifies limits in model-family coverage, uneven cross-task effects, evaluation breadth and judging, and compute-constrained experimentation. These boundaries qualify how broadly the results can be generalized and validated.

Limitations

  • Final experiments use Qwen models; cross-family generalization remains to be tested with the completed adaptation recipes.Family selection used preliminary experiments, but the authors call for retesting across families now that full recipes are built.Source: Limitations · p. 7
  • Adaptation can change English, French, instruction-following, and tool-use performance unevenly, especially for the smallest models, so deployments need task-specific validation.The authors state this validation need for multilingual or task-specific deployments.Source: Limitations · p. 7
  • The evaluation suite could be expanded to cover industrial failure modes, agentic scenarios, larger user studies, and Senegal-specific safety tests.Examples of industrial failure modes include adversarial inputs, privacy leakage, ASR noise, and low-literacy users.Source: Limitations · p. 7
  • LLM-as-a-judge results for Wolof warrant particular caution, and the authors identify judging as an area for improvement despite human assessments.Source: Limitations · p. 7
  • Single-node training on one or two GPUs constrained ablation breadth and multi-seed runs, although the authors consider task diversity a counterbalance.Source: Limitations · p. 7

Ethics and Societal Impact

The paper reports ethical practices and risks, describes its data and validation resources, and documents training and evaluation infrastructure. It also qualifies the model-size comparison by noting that commercial-system parameter ranges are indicative estimates.

Ethics and Societal Impact

  • Annotators were company employees and paid; public datasets had permissive licenses, and LLMs assisted with repetitive LaTeX structures and English editing.Source: Ethics and Societal Impact · p. 7
  • Translated and web-scraped data may reinforce gender, ethnic, or regional bias; the authors call for Wolof-specific fairness audits and alignment, culturally relevant red-teaming, and human escalation for complex service cases.They advocate human-in-the-loop workflows to preserve employment while models handle routine customer-service queries.Source: Ethics and Societal Impact · p. 7

Appendix Contents

  • The appendix lists details on aggregate scores, instruction tuning, preference alignment, human assessment, developmental and stage-by-stage results, and task-level results.Source: Appendix Contents · p. 7

A Overall Score as a Function of Model Size

  • Figure 1 plots aggregate score against model size, distinguishing adapted models, unadapted baselines, Wolof-aware literature models, and commercial APIs.Commercial API parameter ranges are prospectively estimated from unofficial online information, not vendor-confirmed specifications.Source: A Overall Score as a Function of Model… · p. 12, Figure 1 · p. 13

B.1 Data

  • Table 5 reports training, validation, and test statistics by language and task; Wolof-containing data make up approximately 90% of training examples.Source: §B.1 · p. 14, Table 5 · p. 15

B.1.1 Datasets

  • The dataset mixture combines bilingual French–Wolof text with translated instruction sources, including Alpaca, Dolly-15k, and SAMSum, plus Wolof dialogue and customer-care data.Alpaca broadens single-turn instruction coverage, Dolly-15k contributes diverse human-authored prompts, and SAMSum adds dialogue-summarization supervision.Source: §B.1.1 · p. 14

B.1.2 Validation data

  • Validation covers bilingual and translated sets, English and French conversation, Wolof dialogue, and structured output and tool use; a composite score guides checkpoint selection.Source: §B.1.2 · p. 15

B.2 Training Setup

  • SFT training used three epochs, paged AdamW, a cosine schedule, and a peak learning rate of 2×10−5, with gradient clipping and weight decay.The reported setup also specifies a 5% warmup, gradient clipping at 1.0, and weight decay of 0.1.Source: §B.2 · p. 15

C.1 Data

Dolci-Instruct-DPO contains 259,784 preference pairs, split into 233,804 training and 25,980 validation pairs. Its data spans instruction following, knowledge, coding, mathematics, safety, multilingual tasks, and other categories.

C.1 Data

  • 259,784 preference pairs are split into 233,804 training and 25,980 validation pairs, stratified across sources.Table 7 reports the corpus composition and Qwen3 token-length statistics.Source: §C.1 · p. 16, Table 7 · p. 17
  • 72,296 pairs (27.8%) cover instruction following through real-world queries, annotated dialogues, and synthetic constraint-following tasks.Sources include Chat, WildChat, OpenAssistant Conversations, and synthetic tasks with verifiable constraints.Source: §C.1 · p. 16
  • 57,319 pairs (22.1%) provide supervision for factual correctness, multi-step explanations, and domain-specific knowledge.The partition includes scientific information extraction, reasoning traces, and preference comparisons scored by GPT-4.Source: §C.1 · p. 16
  • 41,732 pairs (16.1%) target coding, while 27,045 pairs (10.4%) target mathematics.Coding examples include program summaries and algorithmic problems; math pairs rank correct answers and valid steps above hallucinated paths.Source: §C.1 · p. 16
  • 26,706 pairs (10.3%) support safety alignment, teaching refusal of dangerous requests while retaining helpful answers to safe queries.The multilingual partition contains 6,500 pairs (2.5%) from Aya; Wolof is absent, and the data is intended to help retain multilingual ability.Source: §C.1 · pp. 16–17
  • Long-context reasoning contains 9,975 pairs (3.8%), with inputs averaging 773.7 tokens and chosen answers averaging 177.9 tokens.The remaining 18,211 pairs (7.0%) cover tables, classic NLP tasks, and general questions, including data extraction and format-following.Source: §C.1 · pp. 17–18

C.2 Training Setup

The training and evaluation setup combines model-size-specific compute and shared hyperparameters with human checks of translated data and Wolof judge reliability. Developmental experiments also compare training-data ablations.

C.2 Training Setup

  • Training used either 2× NVIDIA H100 80GB GPUs for Qwen3-14B or 1× H200 140GB for 8B models, with effective batch size 512 and validation-loss checkpoint selection.Training lasted about one day for 0.6B–1.7B models and up to eight days for the largest model; tuning selected β = 0.2 and peak learning rate 5×10−6.Source: §C.2 · p. 18, Table 8 · p. 18

D.1 Translation Quality

  • Google Translate outputs scored 4.59/5 for semantic adequacy and 4.78/5 for Wolof quality across 50 sampled FLORES-200 sentence pairs.Sentence-level METEOR correlated weakly with human adequacy ratings (r = 0.12), and the translations were retained as documented silver data.Source: §D.1 · pp. 18–19

D.2 LLM-as-a-Judge Reliability

  • Wolof speakers agreed with Gemini judge scores on 95% of 450 predictions for Gemini-2.5-flash and 99% for Gemini-2.5-pro.Three speakers validated the models on held-out data.Source: §D.2 · p. 19

E Developmental results

  • Developmental results are reported using the same normalized skill aggregates as the main automatic evaluation.The supplied passage describes the reporting basis but does not state developmental outcomes.Source: E Developmental results · p. 20

E.1 Instruction Ablation

  • The instruction ablation compares Qwen2.5-7B-Instruct variants isolating instruction tuning, function-calling data, conversational data, and their combined recipe.The supplied passages identify the comparison and Table 9 but do not report its outcome.Source: §E.1 · p. 20, Table 9 · p. 20

E.2 Continued Pre-training

The appendix examines whether continued pre-training or vocabulary augmentation improves a Qwen2.5-7B-Instruct adaptation setup. The supplied passages describe the comparisons but do not provide their outcomes.

E.2 Continued Pre-training

  • Continued pre-training plus SFT is compared with the Qwen2.5-7B baseline and SFT-only recipe to test whether raw Wolof text justifies a CPT stage.Table 10 reports normalized scores and ranks, with higher scores and lower ranks being better.Source: §E.2 · p. 20, Table 10 · p. 20

E.3 Tokenization

  • A tokenizer-extension variant is compared with the Qwen2.5-7B instruction-tuned setup to test vocabulary adaptation and broader skill preservation.Table 11 reports normalized scores and ranks, with higher scores and lower ranks being better.Source: §E.3 · p. 20, Table 11 · p. 20

F Stage-by-Stage Results

Post-SFT recovery strategies vary with model capacity: DPO and merging have different effects across sizes, and the appendix compares these variants with Qwen3 baselines and adapted models.

F Stage-by-Stage Results

  • Only SFT uses Wolof data; DPO and merging aim to recover general capabilities while retaining Wolof skills acquired during SFT.Table 12 compares Qwen3 baselines, adapted models, and DPO and model-merging variants; merging uses a 40/60 interpolation between unadapted and adapted checkpoints.Source: F Stage-by-Stage Results · p. 21, Table 12 · p. 21
  • At 14B, merging contributes more than DPO after SFT; at 8B, DPO and merging tie individually and combine for a smaller gain, while smaller models show weaker or unstable recovery.At 4B and 1.7B, DPO is the strongest post-SFT option and merging becomes unstable; at 0.6B, neither stage improves on SFT.Source: F Stage-by-Stage Results · p. 21, Table 12 · p. 21

G Detailed Final Results

The detailed results show gains over the respective base models across nearly all public Wolof tasks for the adapted 4B, 8B, and 14B models. Tables report normalized scores and ranks across task-specific result groups.

G Detailed Final Results

  • The adapted 4B and 8B outperform their respective bases on all 24 public Wolof tasks, and the 14B does so on 23.The appendix reports that aggregate Wolof gains are not driven by a small subset of benchmarks.Source: G Detailed Final Results · p. 22
  • The listed Wolof task groups include named-entity recognition, FAQ question answering, translation, and cultural question answering.These examples appear among the appendix's per-task listings.Source: G Detailed Final Results · p. 23
  • Detailed Wolof understanding and generation tables report normalized scores, where higher is better, and ranks, where lower is better.The tables use colors to indicate ascending position within column ranges.Source: Table 13 · p. 23, Table 14 · p. 24

H Detailed tasks

This appendix details the evaluation suite across Wolof-specific and multilingual tasks, including customer care, translation, reasoning, instruction following, safety, dialogue, and banking and speech applications. It specifies datasets, sample counts, task inputs and outputs, and reporting metrics where provided, while distinguishing reported from diagnostic metrics and noting that some proprietary customer-service data are not released.

H Detailed tasks

  • Evaluation descriptions specify each task’s split, subset, language, sample count, and reported metric; proprietary customer-service datasets are described only at task level and are not released.When multiple metrics are computed, the paper reports one designated metric, using it as the aggregate task score when applicable; other metrics are logged for diagnostics and analysis.Source: H Detailed tasks · p. 38

H.1 afrifact_claim_classification

  • AfrIFact tasks evaluate Wolof claim verification and evidence-span extraction, using the Wolof subset with 1,658 samples for each task.Claim verification reports macro_f1; evidence extraction reports partial_f1 and returns minimal spans copied from the document. The cited task paper is (Azime et al., 2026).Source: §H.1 · p. 38, §H.2 · p. 38

H.3 arc_challenge_reasoning

  • Broader evaluations include ARC Challenge reasoning, demographic-bias questions, and single-turn multilingual instruction following, with 1,172, 500, and 3,641 samples, respectively.Their reported metrics are accuracy, accuracy, and micro_accuracy, respectively; Multi-IF evaluation considers only the first turn. The cited sources are (Clark et al., 2018), (Parrish et al., 2022), and.Source: §H.3 · p. 38, §H.9 · p. 40, §H.10 · p. 40

H.4 aya_wolof_culture_qa

  • Aya evaluations cover Wolof cultural QA, POS tagging, and orthographic correction on test sets of 865, 159, and 244 samples, respectively.Their reported metrics are llm_overall, token_accuracy, and char_edit_distance, respectively; the tasks cite (Singh et al., 2024a).Source: §H.4 · p. 38, §H.5 · p. 39, §H.6 · p. 39

H.7 aya_wolof_translation_from_wol

  • Aya translation tasks evaluate Wolof-to-English-or-French translation on 16 samples and English-or-French-to-Wolof translation on 31 samples, reporting meteor for both.Both use the Aya dataset’s test split and cite (Singh et al., 2024a).Source: §H.7 · p. 39, §H.8 · p. 39

H.16 corporate_faq_wolof_long_context_- mrqa

  • FAQ question answering includes a conversational prompt variant, Wolof document QA, and long-context QA with distractors; the latter uses 99 samples and both document-QA tasks report METEOR.Standard document QA uses 122 samples; long-context QA combines FAQ documents with Wolof distractors and uses 99 samples.Source: §H.16 · pp. 41–42, §H.15 · p. 41, §H.17 · p. 42

H.18 corporate_conversation_coherence

  • Conversation coherence evaluates context awareness, fluency, and Wolof/French language respect, while dialog-act labeling classifies acts for each turn.The tasks use 287 and 149 samples respectively, reporting overall_coherence and macro_f1.Source: §H.18 · p. 42, §H.19 · p. 42

H.20 corporate_full_analysis

  • Full conversation analysis produces summaries, entities, and turn-level sentiment and dialog acts, while GDPR labeling extracts personal data; both use 150 samples.Full analysis reports overall_score; GDPR labeling reports exact_f1.Source: §H.20 · p. 42, §H.21 · p. 43

H.22 corporate_ner

  • The call-center suite separately evaluates public/general named-entity extraction and dialogue-level resolution classification, reporting exact_f1 and macro_f1.Both tasks use 149 or 150 samples: NER uses 150 and resolution classification uses 149.Source: §H.22 · p. 43, §H.23 · p. 43

H.24 corporate_sentiment_- classification

  • Turn-level sentiment classification and Wolof conversation summarization use 149 and 150 samples, reporting macro_f1 and llm_overall respectively.Summarization's overall score averages semantic and Wolof-language judge scores; ROUGE and METEOR are also computed.Source: §H.24 · pp. 43–44, §H.25 · p. 44

H.26 flores_translate_eng_to_wol

  • FLORES devtest evaluates English-to-Wolof translation on 1,012 samples using METEOR, with BLEU logged for diagnostics.The task uses the FLORES+ dataset and takes topic, domain, and English sentence as inputs.Source: §H.26 · p. 44

H.28 flores_translate_wol_to_eng

  • The reverse FLORES direction evaluates Wolof-to-English translation on 1,012 devtest samples, reporting METEOR and logging BLEU and an LLM score.Inputs include topic, domain, and a Wolof sentence.Source: §H.28 · p. 44

H.29 flores_translate_wol_to_fra; H.30 flores_translation

  • FLORES translation covers Wolof-to-French on 5 devtest samples and language-pair translation on 1,012 samples, reporting METEOR for both tasks.BLEU is also logged diagnostically; the general task supports language pairs from FLORES+.Source: §H.29 · p. 45, §H.30 · p. 45

H.31 gpqa_english_mcqa; H.32 gpqa_french_mcqa

  • English and French GPQA Diamond tasks evaluate graduate-level scientific reasoning on test sets of 198 and 197 samples, respectively, using accuracy.The French task uses a French translation of the benchmark; both tasks provide question, choices, and choice labels.Source: §H.31 · p. 46, §H.32 · p. 46

H.33 gsm8k_reasoning

  • GSM8K evaluates grade-school math word-problem solving on 1,319 test samples, reporting accuracy from numerical answers.The description characterizes the problems as linguistically diverse and involving basic arithmetic.Source: §H.33 · p. 46

H.34 halueval_qa_hallucination_- detection

  • HaluEval tests hallucination detection by asking models to identify which of two QA responses contains fabricated or incorrect information, reporting accuracy on 500 samples.F1 is also logged diagnostically.Source: §H.34 · pp. 46–47

H.35 humaneval_code_generation

  • HumanEval evaluates code generation on 164 test prompts using pass_at_1 as its reporting metric.Syntax-error and runtime-error rates are logged for diagnostics.Source: §H.35 · p. 47

H.36 ifbench_instruction_following

  • IFBench evaluates English instruction-following generalization to verifiable out-of-domain constraints on 300 test prompts, reporting accuracy.Macro-accuracy is also logged diagnostically.Source: §H.36 · p. 47

H.37 irokobench_mgsm

  • IrokoBench MGSM evaluates grade-school math on 250 test samples, with a prompt directing the assistant to answer in Wolof and exact_match as the reporting metric.MAE and MSE are logged diagnostically.Source: §H.37 · p. 47

H.38 irokobench_mmlu

  • IrokoBench MMLU evaluates four-choice question answering using subject, question, and choices on 500 test samples, reporting accuracy.Source: §H.38 · p. 47

H.38 irokobench_mmlu; H.39 irokobench_xnli; H.40 masakha_ner

  • Wolof language tasks include subject-based multiple-choice questions, three-label premise–hypothesis inference on 600 samples, and news named-entity extraction on 536 samples.The inference task reports accuracy; named-entity extraction reports exact_f1.Source: §H.38 · p. 48, §H.39 · p. 48, §H.40 · p. 48

H.41 mifeval_instruction_following; H.44 multi_if_instruction_following

  • Instruction-following tests assess verifiable constraints in English and French, using M-IFEval’s 541 English and 235 French samples and Multi-IF’s 808 samples.The reporting metrics are strict_prompt_accuracy for M-IFEval and accuracy for Multi-IF.Source: §H.41 · p. 48, §H.44 · p. 49

H.42 mmlu_mcqa_english; H.43 mmlu_mcqa_french

  • English and French academic-knowledge evaluations use the MMMLU test split, with 14,042 samples per language and accuracy as the reporting metric.The English benchmark spans 57 academic and professional subjects; the French task uses the French split of the multilingual release.Source: §H.42 · pp. 48–49, §H.43 · p. 49

H.45 personachat_conversation_- coherence; H.48 webnlg_qa_conversational_mrqa

  • Conversation tasks assess English persona consistency and context awareness across turns, alongside conversational question answering from WebNLG-QA documents and history.PersonaChat uses 500 validation samples and reports overall_coherence_llm; WebNLG-QA uses 232 challenge samples and reports llm_good.Source: §H.45 · p. 49, §H.48 · p. 50

H.46 ruler_mrqa; H.47 toxigen_implicit_hate_detection

  • The suite includes long-context retrieval and reasoning over RULER synthetic tasks, plus detection of implicit hate or toxicity in 940 samples.RULER covers needle-in-a-haystack variations, variable tracking, and common-word extraction; the toxicity task reports accuracy.Source: §H.46 · pp. 49–50, §H.47 · p. 50

H.49 wildjailbreak_harmful_request_- refusal

  • A separate refusal evaluation tests whether models reject harmful requests without over-refusing benign ones, using WildJailbreak methodology and LLM-as-judge accuracy.It evaluates 500 samples and also logs attack-success and over-refusal rates.Source: §H.49 · p. 50

H.50 wolbanking77_audio_intent_- classification; H.52 wolbanking77_intent_- classification

  • Banking-query evaluation covers 10 Wolof intent categories from clean reference or ASR transcriptions and a separate 77-category classification task supporting Wolof, French, and English.The 10-category evaluation uses 641 samples and reports macro_f1; the 77-category task description gives no sample count or metric in the supplied passage.Source: §H.50 · p. 51, §H.52 · p. 51

H.51 wolbanking77_audio_transcription_- correction

  • Wolof speech evaluation also tests correction of real ASR transcriptions toward clean references, using 641 samples and mean relative WER difference as its reporting metric.The task also logs mean WER before and after correction.Source: §H.51 · p. 51

H.52 wolbanking77_intent_- classification

  • Intent classification uses 500 WolBanking77 test samples, with macro_f1 as the reporting metric and utterances as system inputs.Accuracy and micro_f1 are logged for diagnostic analysis.Source: §H.52 · p. 51

H.53 wolbanking77_translation

  • Banking-query translation covers Wolof, French, and English, evaluates 500 WolBanking77 test samples, and reports meteor.BLEU is also computed for diagnostics, and LLM-as-judge is optionally used when the target is not Wolof.Source: §H.53 · p. 52

Related papers

Loading 2610.20824v1…