PaperlayerView exact revision on arXiv

Source-linked AI summary

AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages

Kosei Uemura, Miaoran Zhang, David Ifeoluwa Adelani

arXiv:2510.23896v2cs.CL

TL;DR

African-language embedding quality lacks task-diverse, systematically comparable evaluation because existing multilingual benchmarks provide sparse and uneven coverage. The paper introduces AfriMTEB and AfriMTEB-Lite, and adapts mE5 through cross-lingual contrastive distillation as AfriE5. AfriE5 achieves the strongest overall macro-average among open-weight models on both suites while remaining competitive with Gemini Embedding-001.

  • Problem

    African languages remain underrepresented in multilingual embedding benchmarks, leaving their quality across tasks and languages insufficiently standardized and understood.

    Source: §1 · p. 1, §6 · p. 9

  • Method

    The paper builds AfriMTEB and its uniform nine-language Lite suite, then adapts mE5-Large-Instruct to African languages through cross-lingual contrastive distillation.

    Source: §1 · pp. 1–2

  • Results

    AfriE5 achieves the strongest overall macro-average on AfriMTEB-Full and AfriMTEB-Lite, with statistically significant gains on several task families and competitiveness with Gemini Embedding-001.

    Source: §1 · p. 2, §4.1 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Targeted adaptation on a carefully selected subset of African languages can transfer to a substantially larger benchmark while preserving the backbone’s general utility.

    Source: §1 · p. 2, §7 · p. 9

    Coverage remains incomplete across African languages, dialects, orthographies, registers, domains, and evaluation dimensions, while machine translation, automatic quality estimation, and single-teacher distillation may introduce bias.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Text embeddings are an essential building component of several NLP tasks such as retrieval-augmented generation which is crucial for preventing hallucinations in LLMs. Despite the recent release of massively multilingual MTEB (MMTEB), African languages remain underrepresented, with existing tasks often repurposed from translation benchmarks such as FLORES clustering or SIB-200. In this paper, we introduce AfriMTEB -- a regional expansion of MMTEB covering 59 languages, 14 tasks, and 38 datasets, including six newly added datasets. Unlike many MMTEB datasets that include fewer than five languages, the new additions span 14 to 56 African languages and introduce entirely new tasks, such as hate speech detection, intent detection, and emotion classification, which were not previously covered. Complementing this, we present AfriE5, an adaptation of the instruction-tuned mE5 model to African languages through cross-lingual contrastive distillation. Our evaluation shows that AfriE5 achieves state-of-the-art performance, outperforming strong baselines such as Gemini-Embeddings and mE5.

1 Introduction

African-language embedding quality remains difficult to assess because existing benchmarks provide sparse, uneven, and often translation-derived coverage. The paper addresses this gap with AfriMTEB, AfriMTEB-Lite, and AfriE5, achieving the strongest reported benchmark performance.

1 Introduction

  • AfriMTEB addresses underrepresented African languages with a standardized evaluation spanning diverse tasks and application settings.Its Full suite covers 59 languages and 38 datasets across bitext mining, classification, semantic similarity, retrieval, clustering, and reranking.Source: §1 · pp. 1–2
  • AfriMTEB-Lite provides controlled comparisons across nine geographically and typologically diverse African languages with near-uniform task coverage.The compact suite supports language-, task-, resource-level, ablation, and statistical analyses that uneven multilingual benchmarks make difficult.Source: §1 · p. 2, Figure 2 · p. 2
  • AfriE5 adapts mE5-Large-Instruct through cross-lingual contrastive distillation using translated and automatically filtered MNLI/SNLI data plus teacher scores.The training data is expanded across multiple source and target directions to encourage cross-lingual alignment.Source: §1 · p. 2
  • 63.7 average on AfriMTEB-Lite and 62.4 on AfriMTEB-Full put AfriE5-Large-Instruct ahead of mE5-large-instruct and Gemini embedding-001.The reported comparisons are 63.7 versus 62.0 and 63.1 on Lite, and 62.4 versus 61.3 and 60.6 on Full.Source: Figure 1 · p. 1, §1 · p. 2

2 AfriMTEB Benchmark

AfriMTEB is a region-specific benchmark for evaluating African-language text embeddings across a broad range of tasks. The supplied overview identifies its task-oriented benchmark framing but provides no further scope details.

2 AfriMTEB Benchmark

  • AfriMTEB is designed to evaluate text embeddings for African languages across a broad range of tasks.Source: §2 · p. 2
  • The benchmark overview labels its organization by task, task family, datasets, and number of languages.Source: §2 · p. 3

MMTEB AfriMTEB

AfriMTEB extends MMTEB with broader and more task-diverse African-language coverage, including six new datasets. AfriMTEB-Lite then enforces uniform nine-language coverage for controlled comparison and analysis.

MMTEB AfriMTEB

  • AfriMTEB covers 59 languages, 14 tasks, and 38 datasets, filling MMTEB gaps in hate speech, intent, and multi-label emotion evaluation.The benchmark adds six datasets and expands beyond sparse or uneven African-language representation in existing multilingual evaluations.Source: MMTEB AfriMTEB · p. 3, Table 1 · p. 3

2.1 Existing MMTEB Datasets

  • Existing MMTEB-derived resources include bitext mining, pair classification, topic and sentiment classification, semantic similarity, retrieval, clustering, and reranking, with uneven language coverage.AfriMTEB computes macro-averages within tasks and then across tasks and families so one language or task cannot dominate the overall score.Source: §2.1 · p. 3, MMTEB AfriMTEB · p. 3

2.2 New Datasets in AfriMTEB

  • The six additions broaden task and language coverage through AfriXNLI, EmotionAnalysisPlus, AfriHate, InjongoIntent, KinNews, and SIB200_14Classes.They extend entailment to 15 African languages, add multi-label emotion coverage for 14, hate speech for 14, intent detection for 16, and Kinyarwanda news classification.Source: §2.2 · p. 4
  • SIB200_14Classes increases classification difficulty by merging fine-grained topics into 14 broader categories while retaining 56 African languages.The broader labels increase intra-class diversity and strengthen cross-lingual assessment.Source: §2.2 · p. 4

2.3 AfriMTEB-Lite Construction

  • AfriMTEB-Lite retains 13 datasets covering the same nine African languages to reduce variance and enable controlled comparisons, confidence intervals, and reproducible ablations.The selected suite spans classification, retrieval, bitext mining, clustering, pair classification, and multi-label classification.Source: §2.3 · p. 4

3 Adapting Embedding Models to African Languages

This section presents AfriE5-Large-Instruct, adapting mE5 through cross-lingual contrastive distillation and evaluating it with AfriMTEB. The method combines multilingual training data construction, hard negatives, teacher scores, and controlled experimental settings.

3 Adapting Embedding Models to African Languages

  • AfriE5-Large-Instruct adapts mE5 using cross-lingual contrastive distillation to address performance gaps in African-language embeddings.The model is evaluated systematically under AfriMTEB.Source: §3 · p. 4

3.1 Method

  • The unified objective combines contrastive learning with knowledge distillation over batches of queries and associated passages.Contrastive learning uses query–passage similarities, while distillation aligns student predictions with teacher reranker scores.Source: §3.1 · p. 5

3.2 Cross-lingual Training Data Construction

  • Cross-lingual training data translates MultiNLI and SNLI into nine African languages, filters translations below a 0.75 quality threshold, and expands examples across language directions.The construction also uses hard-negative mining to retrieve semantically similar but irrelevant passages for fine-grained discrimination.Source: §3.2 · pp. 5–6

3.3 Experimental Settings

  • Training fine-tunes mE5-Large-Instruct for one epoch with cross-device negatives and BGE Reranker v2 m3 teacher scores on a single GPU.Samples within each batch come from the same dataset to maintain consistent supervision.Source: §3.3 · p. 6

4 Results

AfriE5-Large-Instruct leads overall AfriMTEB performance, with gains extending across 59 languages despite training on nine. Results vary by task family, with Gemini strongest on several classification and similarity tasks.

4.1 AfriMTEB Results

  • 62.4 macro average places AfriE5-Large-Instruct ahead of Gemini embedding (60.6) and mE5-Large-Instruct (61.3) on AfriMTEB-Full.The overall improvement over mE5-Large-Instruct is statistically significant.Source: §4.1 · p. 7
  • E5 variants reach about 85 on bitext mining and AfriE5-Large-Instruct leads clustering at 62.9, while Gemini scores 72.2 on bitext mining.These results favor models with robust cross-lingual alignment.Source: §4.1 · p. 7
  • 64.0 reranking gives AfriE5-Large-Instruct an advantage over mE5-Large-Instruct (61.9) and Gemini embedding 001 (63.4).The training recipe combines cross-lingual contrastive supervision with BGE-M3 cross-encoder distillation.Source: §4.1 · p. 7
  • Gemini embedding 001 leads single-label classification at 50.0, multi-label classification at 32.7, pair classification at 71.6, retrieval at 77.5, and STS at 65.0.Its strongest results are concentrated in classification-style and sentence-level similarity tasks.Source: §4.1 · p. 7
  • AfriE5-Large-Instruct reaches 62.4 across 59 languages despite supervision centered on nine, with significant gains in pair classification, reranking, retrieval, and multi-label classification.It trades small losses on bitext mining, single-label classification, and STS for these family-level gains.Source: §4.1 · p. 7

4.2 AfriMTEB-Lite Results

  • AfriE5-Large-Instruct scores 63.7 on AfriMTEB-Lite, exceeding mE5-Large-Instruct (62.0) and Gemini embedding (63.1) across nine languages and twelve tasks.Paired tests find significant improvements on 9 of 12 tasks, while multi-seed variation stays within ±0.1.Source: §4.2 · p. 7
  • AfriE5 achieves the highest language-level macro average on six of nine Lite languages, including Swahili, Amharic, Xhosa, Igbo, Yoruba, and Oromo.The margins are particularly pronounced for lower-resource Oromo and Xhosa.Source: §4.2 · p. 8
  • AfriE5 improves AfriSenti by +3.7, AfriXNLI by +4.5, SIB200clustering by +1.8, and SIB200_14Classes by +4.4 over mE5-Large-Instruct.The largest gain is on SIB200_14Classes, rising from 21.8 to 26.2.Source: §4.2 · p. 8

5 Ablation

The ablations isolate cross-lingual dataset expansion and translation-quality filtering as important AfriE5 training choices. Performance peaks when expansion is enabled and filtering balances translation quality with retained data.

Impact of cross-lingual dataset expansion

  • Cross-lingual dataset expansion raises the average from 62.3 to 63.2 when other training settings are fixed.Expansion adds target–target, English–English, target–English, and English–target configurations.Source: §5 · p. 8

Machine translation quality threshold (QE)

  • A 0.75 SSA-COMET-MTL quality threshold yields 63.2, outperforming 0.67 at 62.5 and 0.80 at 58.3.The middle threshold retains around 60K samples, balancing translation quality and coverage; 0.67 retains noisy data and 0.80 leaves only 7,500 samples.Source: §5 · pp. 8–9

Ablation scope

  • The ablation excludes Belebele retrieval because that dataset was added after the study was finalized.Table 4 therefore does not report Belebele results.Source: §5 · p. 8, Table 4 · p. 9

6 Related Work

Prior multilingual embedding benchmarks have expanded regional coverage, but African-language evaluation remains uneven. AfriMTEB follows this regional-benchmark trend while targeting persistent coverage and performance gaps.

Multilingual text embedding benchmarks

  • MMTEB and other regional suites broaden multilingual embedding evaluation across language- and region-specific tracks.Examples include SEA-BED, PL-MTEB, MTEB-French, C-MTEB, JMTEB, and VN-MTEB.Source: §6 · p. 9

Multilingual text embedding models

  • Commercial APIs and open-weight models provide strong multilingual embedding baselines, including Gemini embedding-001, mE5, Qwen3-Embedding, Embedding Gemma, GTE, LaBSE, and BGE-M3.BGE-M3 combines multilingual and multi-function training as a competitive open baseline.Source: §6 · p. 9
  • Despite these advances, African-language coverage and performance remain uneven, motivating region-specific evaluation and targeted adaptation.AfriMTEB addresses this gap with broad task coverage for African languages.Source: §6 · p. 9

7 Conclusion

The paper introduces AfriMTEB as a broad African-language embedding benchmark and AfriE5 as an adapted multilingual embedding model. AfriE5 achieves the strongest overall macro-average among open-weight models while remaining competitive with Gemini Embedding-001.

7 Conclusion

  • AfriMTEB spans 59 languages and 14 tasks, while AfriE5 adapts mE5-large-instruct through cross-lingual contrastive distillation.AfriMTEB provides standardized evaluation, and AfriE5 supplies an adapted embedding model for African languages.Source: §7 · p. 9
  • AfriE5 achieves the strongest overall macro-average among open-weight models on both AfriMTEB-Full and AfriMTEB-Lite.The model also remains competitive with the proprietary Gemini Embedding-001 baseline.Source: §7 · p. 9
  • Ablations identify cross-lingual dataset expansion and balanced translation filtering as crucial for AfriE5’s gains.Source: §7 · p. 9
  • Together, AfriMTEB and AfriE5 provide a standardized evaluation framework and practical reference point for African-language embedding research.Source: §7 · p. 9

Limitations

The evaluation does not cover all African linguistic varieties or all embedding applications, and several data and comparison choices constrain interpretation. These limitations include underrepresented languages, machine-translation and teacher biases, and restricted evaluation scope.

Limitations

  • Many African languages, dialects, orthographies, and code-switched registers remain underrepresented despite AfriMTEB’s 59-language coverage.Source: Limitations · p. 9
  • The adaptation data relies on NLLB-200 translation and SSA-COMET quality estimation, whose reliability varies across language pairs and domains.Source: Limitations · p. 9
  • Distillation from a single BGE Reranker v2 m3 teacher can transfer biases toward languages and styles favored by the translation, quality-estimation, and teacher pipelines.Source: Limitations · p. 9
  • The evaluation covers text-only sentence and paragraph embeddings, macro-averages, and formal or news-heavy domains rather than calibration, robustness, fairness, RAG, multimodal, or long-context settings.Source: Limitations · pp. 9–10
  • Transfer may degrade for typologically distant or extremely low-resource languages, although AfriE5 trained on nine languages generalizes well to 59 in these tests.Source: Limitations · p. 10

A.1 Task Categories

AfriMTEB organizes embedding evaluation into eight task families spanning cross-lingual matching, classification, similarity, clustering, retrieval, and reranking. The families differ in inputs, prediction targets, and how embeddings are used.

A.1 Task Categories

  • Bitext mining retrieves translation pairs between sentence sets in different languages using embedding similarity.Source: §A.1 · p. 13
  • Pair classification predicts a binary or categorical relation between two input sentences from their embeddings.Source: §A.1 · p. 13
  • Single-text classification maps each input to one category, while multi-label classification permits multiple simultaneous labels such as emotions.Source: §A.1 · p. 14
  • Clustering groups texts into clusters corresponding to gold categories using embedding representations and algorithms such as k-means.Source: §A.1 · p. 14
  • Semantic text similarity measures graded semantic relatedness between sentence pairs through embedding similarity.Source: §A.1 · p. 14
  • Retrieval ranks relevant documents for a query, whereas reranking orders a candidate document set by fine-grained relevance.Source: §A.1 · p. 14

A.2 Dataset Descriptions

The appendix describes AfriMTEB’s datasets, baseline models, translation-quality filtering, statistical tests, and training-seed stability. Together, these details define the benchmark’s task coverage and document how AfriE5’s reported gains were evaluated.

A.2.1 Bitext Mining datasets

  • Bitext mining datasets test cross-lingual alignment across professionally translated, religious, informal, community-created, and parallel corpora.FLORES, NTREX, BibleNLP, NollySenti, and Tatoeba provide varied translation styles and domains.Source: §A.2.1 · p. 14

A.2.2 Pair Classification datasets

  • Pair classification uses XNLI and AfriXNLI premise–hypothesis relations to evaluate entailment, contradiction, and neutral-label prediction in African languages.Source: §A.2.2 · p. 14

A.2.3 Classification datasets

  • Classification covers topic, language identification, intent, scenario, news, sentiment, and hate-speech tasks, including African-specific datasets such as InjongoIntent and AfriHate.Source: §A.2.3 · p. 15

A.2.4 Multi-label Classification datasets

  • Multi-label emotion classification allows multiple affective labels per text, unlike categorical sentiment classification.Source: §A.2.4 · pp. 15–16

A.2.5 Semantic Text Similarity datasets

  • Semantic Text Similarity evaluates correlation between embedding similarities and human-annotated graded similarity scores.Source: §A.2.5 · p. 16

A.2.6 Retrieval datasets

  • Retrieval datasets evaluate query-to-passage ranking across reading comprehension, multilingual search, hard negatives, text retrieval, and text-to-image retrieval.Source: §A.2.6 · p. 16

A.4 Detailed Statistics of Machine Translation Quality

  • At a 0.67 SSA-COMET threshold, over 430k translation pairs remain; at 0.75, around 60k remain; and at 0.80, only 7.5k remain.Swahili, Oromo, and Kinyarwanda contribute most pairs, while Igbo and Amharic decline most sharply under stricter filtering.Source: Table 6 · p. 18, §A.4 · p. 18

A.6 Statistical Significance

  • AfriE5 significantly improves over mE5 in 9 of 12 Lite tasks and four of eight Full-suite categories, with a significantly positive Full macro average.Across additional seeds, the Lite macro average varies only from 63.5 to 63.6 versus 63.7 for the main run.Source: §A.6 · p. 19, §A.7 · p. 19

Related papers

Loading 2510.23896v2…