PaperlayerView exact revision on arXiv

Source-linked AI summary

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

arXiv:2608.26941v1cs.CL

TL;DR

Kinyarwanda is poorly served by general multilingual embeddings because it is underrepresented in their training corpora. KinyaEmbed combines KinyaBERT-large with a four-stage contrastive curriculum and checkpoint ensemble, achieving leading STS and clustering results while remaining limited on asymmetric retrieval, bitext retrieval, and non-Wikipedia domains.

  • Problem

    General multilingual encoders perform poorly on Kinyarwanda STS because the language is severely underrepresented in multilingual corpora.

  • Method

    KinyaEmbed uses KinyaBERT-large, four-stage curriculum training with MNRL, and a multi-checkpoint ensemble.

  • Results

    KinyaEmbed reaches Spearman ρ = 0.7298 on SemRel2024-rw, leads Wiki-RW-STS by 8.6%, and has the best document-clustering Silhouette Score.

  • Takeaways & Limitations

    Language-specific pretraining and curriculum training provide STS advantages over general models, while the released model and resources support Kinyarwanda NLP deployment.

  • Takeaways & Limitations

    KinyaEmbed trails retrieval-optimized models on asymmetric IR, has FLORES P@1 = 0.3587, and is evaluated only on Wikipedia-derived text.

Abstract

from arXiv · show

We present KinyaEmbed, the first dedicated sentence embedding model for Kinyarwanda, a morphologically rich Bantu language spoken by over 12 million people in Rwanda. Existing multilingual embedding models such as LaBSE, mE5-large, and OpenAI text-embedding-3-large perform poorly on Kinyarwanda due to severe under-representation in their pre-training corpora. KinyaEmbed is built on KinyaBERT-large and trained via a four-stage curriculum using MultipleNegativesRankingLoss (MNRL): Stage 1 leverages ~18,000 paraphrase pairs from the Official Gazette of Rwanda with three temperature scales; Stage 2 fine-tunes on 715 NLLB-translated MNLI triplets for entailment structure; Stage 3 aligns representations using English-Kinyarwanda OPUS-100 translation pairs; Stage 4 refines with 2,936 high-quality pairs filtered from KinyaCOMET at quality threshold 0.8. We evaluate on SemRel2024-rw and introduce Wiki-RW-STS, a new contamination-free Kinyarwanda STS benchmark of 300 pairs derived from Kinyarwanda Wikipedia. A seven-checkpoint ensemble (all5+23A*2, with the final stage double-weighted) achieves Spearman \r{ho}=0.7298 on SemRel2024-rw, surpassing mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%. KinyaEmbed also achieves the best document clustering silhouette score (0.2146) across all evaluated models. All checkpoints, the KinyaCOMET filtered pairs, and the Wiki-RW-STS benchmark are publicly available.

1. Introduction

Kinyarwanda is severely underserved by NLP infrastructure, and existing multilingual encoders produce poor semantic-similarity embeddings in practice. KinyaEmbed addresses this gap with a dedicated encoder, curriculum training, systematic evaluation, and publicly released resources.

  • Kinyarwanda has almost no dedicated language technology despite being Rwanda’s national language.
  • Existing multilingual encoders nominally support 100+ languages but perform poorly on Kinyarwanda STS embeddings.
  • KinyaEmbed is the first sentence embedding model designed and evaluated specifically for Kinyarwanda.
  • KinyaEmbed uses a four-stage curriculum with MultipleNegativesRankingLoss and a multi-checkpoint ensemble strategy.
  • Spearman ρ = 0.7298 on SemRel2024-rw surpasses all multilingual baselines on Kinyarwanda STS.
  • The work releases 2,936 filtered KinyaCOMET pairs and a 300-pair contamination-free Wiki-RW-STS benchmark.

2. Related Work

Prior multilingual and African-language adaptations broaden coverage, but KinyaEmbed builds on Kinyarwanda-specific pretraining and contrastive curriculum training. Its representation design uses KinyaBERT-large as the sentence-encoding backbone.

  • LaBSE, mE5, and BGE-M3 provide broad multilingual or retrieval-oriented embedding approaches, but treat Kinyarwanda as an incidental tail language.
  • AfroXLMR and AfriE5 improve African-language coverage, while AfriE5 still uses a generic multilingual backbone.
  • KinyaCOMET supplies human quality annotations for Kinyarwanda–English translations that KinyaEmbed repurposes for contrastive training.
  • Curriculum ordering progresses from same-language paraphrases to human-preference translation pairs, while checkpoint averaging combines complementary specializations.
  • Kinyarwanda is heavily agglutinative, and standard BPE tokenizers fragment its words into suboptimal subword units.
  • KinyaBERT-large is a 12-layer transformer pretrained on curated Kinyarwanda Wikipedia, news, legal, and religious documents.

4. Method

KinyaEmbed encodes sentences with normalized KinyaBERT-large representations and trains them through progressively harder data stages using MNRL. Multiple checkpoints are combined to balance monolingual similarity and cross-lingual alignment.

  • 4.1. Sentence Encoding Architecture: Sentence encoding mean-pools final-layer KinyaBERT-large token representations and applies L2 normalization.
  • 4.2. Training Objective: MNRL: MNRL trains anchor–positive pairs while treating every other positive in the batch as an in-batch negative.
  • 4.3. Four-Stage Curriculum Training: Training progresses from easy monolingual paraphrases to hard human-annotated cross-lingual pairs.
  • 4.3. Four-Stage Curriculum Training: Stage 1 uses Official Gazette paraphrases and produces checkpoints sc30, sc35, and sc40.
  • 4.3. Four-Stage Curriculum Training: Stage 2 uses machine-translated MNLI triplets for semantic reasoning, while Stage 3 uses OPUS-100 pairs to align English and Kinyarwanda spaces.
  • 4.3. Four-Stage Curriculum Training: Stage 4 fine-tunes on 2,936 KinyaCOMET pairs with quality score ≥0.8, improving cross-lingual alignment while introducing a modest STS trade-off.
  • 4.4. Ensemble Construction: The ensemble averages seven checkpoints, double-weighting step23A; STS reaches ρ = 0.7298 and FLORES P@1 reaches 0.3587.

5. KinyaCOMET Training Resource

KinyaCOMET human quality annotations are repurposed as a high-quality semantic-equivalence resource for contrastive embedding training. The filtered pairs are released for community use.

  • KinyaCOMET pairs with quality score ≥0.8 are repurposed as semantic equivalents for MNRL training.
  • 2,936 pairs are retained from 4,323 annotated pairs, representing 67.9% of the resource.
  • The filtered pairs are released as a community resource through the KinyaEmbed Hugging Face repository.

6. Experimental Setup

The experiments evaluate KinyaEmbed’s curriculum training across semantic relatedness, bitext mining, retrieval, clustering, and classification, using both established and newly constructed Kinyarwanda benchmarks. Comparisons include multilingual baselines and contamination-controlled evaluation.

  • Training configuration: All four stages use MultipleNegativesRankingLoss with KinyaBERT-large, with temperature, batch size, epochs, and learning rate configured across stages.The training configuration is summarized per stage; all stages use AdamW with learning rate 2 × 10^-5 and 10% warmup.
  • Evaluation benchmarks: SemRel2024-rw contains 222 sentence pairs scored from 0–1 and evaluated using Spearman ρ between cosine similarities and human relatedness scores.It is the official test split of the SemRel2024 Kinyarwanda semantic relatedness task.
  • Evaluation benchmarks: The evaluation covers OPUS-100 and FLORES-200 bidirectional bitext mining using averaged P@1 on English–Kinyarwanda test splits.OPUS-100 uses professional human translations, while FLORES-200 covers diverse domains.
  • Evaluation benchmarks: Wiki-RW-STS contains 300 Kinyarwanda Wikipedia pairs across high, medium, and low similarity levels, with disagreement-filtered native-speaker annotations and no training-data overlap.The benchmark is designed as a contamination-free evaluation set.
  • Downstream tasks: Downstream evaluation includes title-to-body information retrieval on 300 Wikipedia articles, K-means clustering across eight topics, and prototype-based zero-shot classification.Retrieval uses P@1, clustering uses Silhouette Score and Davies-Bouldin Index, and classification uses Top-1 accuracy on 36 labeled articles.
  • Comparative evaluation: Table 2 reports SemRel2024-rw Spearman ρ alongside OPUS-100 and FLORES-200 P@1, while Table 3 reports fair Wiki-RW-STS results on 300 unseen pairs.The tables compare KinyaEmbed with multilingual baselines on complementary monolingual and cross-lingual tasks.

7. Results

KinyaEmbed achieves the strongest Kinyarwanda semantic-similarity and clustering results, while retrieval-optimized models remain better at title-to-body information retrieval. Its ensemble combines strong STS and cross-lingual performance, whereas classification results are noisy on the small evaluation subset.

  • ρ = 0.7298 on SemRel2024-rw is the highest STS score, exceeding mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%.
  • KinyaEmbed wins document clustering on both Silhouette and Davies-Bouldin metrics, while retrieval-optimized models perform better on information retrieval.Table 4 evaluates Wikipedia title→body IR, K-means clustering, and zero-shot classification.
  • Classification scores are noisy because evaluation uses only 36 of 300 articles, so the authors draw no strong conclusions from this task.
  • KinyaEmbed reaches Silhouette 0.2146, ahead of LaBSE at 0.1882, mE5-large-instruct at 0.1073, and BGE-M3 at 0.1086.
  • The all5+23A×2 ensemble yields a 25.8% relative FLORES gain at a small STS cost, producing a Pareto improvement over single checkpoints.Double-weighting step23A changes STS from 0.7395 to 0.7298 while increasing FLORES P@1 from 0.2851 to 0.3587.

8. Analysis

Stage 1 gazette paraphrase training provides the largest STS improvement, while later cross-lingual stages trade some monolingual STS for better alignment. Checkpoint ensembling recovers this trade-off, and representation analysis links multilingual-model underperformance to similarity collapse.

  • 8.1. Stage-by-Stage Score Progression: Stage 2 maintains STS at 0.734 while its MNLI triplets sharpen entailment and contradiction boundary discrimination.
  • 8.1. Stage-by-Stage Score Progression: Stages 3 and 4 reduce monolingual STS from 0.713 to 0.710 while increasing FLORES P@1 from 0.271 to 0.359.The passages describe this as a trade-off between monolingual semantic similarity and cross-lingual translation alignment.
  • 8.1. Stage-by-Stage Score Progression: +0.359 over KinyaBERT-large is the largest single SemRel2024-rw gain, delivered by Stage 1 gazette paraphrase training.
  • 8.1. Stage-by-Stage Score Progression: The all5+23A×2 ensemble recovers STS to 0.730 while maintaining the FLORES improvement over late-stage checkpoints.
  • mE5-large similarities cluster in [0.82, 0.96], whereas KinyaEmbed spans [0.15, 0.98] and correlates with human relatedness at Pearson 0.71.
  • KinyaBERT-specific pretraining prevents multilingual representation collapse, while contrastive fine-tuning calibrates embeddings to human relatedness judgments.

9. Discussion

KinyaEmbed’s language-specific pretraining yields strong semantic-similarity and clustering performance, while retrieval remains a relative weakness. The authors identify asymmetric retrieval, cross-lingual alignment, and domain coverage as key boundaries.

  • Why language-specific pretraining wins: 20.9% STS advantage over mE5-large is attributed to KinyaBERT-large’s Kinyarwanda-specific pretraining.The discussion argues that generic multilingual capacity does not adequately represent Kinyarwanda morphology and vocabulary.
  • Task-modality split: KinyaEmbed clusters best but underperforms retrieval-optimized models on asymmetric information retrieval.The authors frame this as a task-modality split between semantic similarity and asymmetric retrieval.
  • Limitations: The authors identify asymmetric information retrieval as a limitation and propose retrieval instruction-tuning as a natural extension.This constrains KinyaEmbed’s use for applications where query and document roles are asymmetric.
  • Limitations: FLORES P@1 is 0.3587, below bitext-specialized models scoring 0.98–1.00, because 2,936 pairs cannot match billions of translation pairs for alignment.This limitation specifically concerns cross-lingual alignment rather than monolingual semantic similarity.
  • Limitations: Evaluation is limited to Wikipedia-derived text, leaving health, legal, and agricultural domains in need of domain-specific assessment.The stated boundary is evaluation scope, not a claim that the model fails in those domains.
  • Practical impact: The model is CPU-deployable without an API dependency, supporting semantic search, document clustering, and cross-lingual retrieval in resource-limited environments.The discussion names Wikipedia search, government and health documents, and English–Kinyarwanda retrieval as practical applications.

10. Conclusion

The paper presents KinyaEmbed as a dedicated Kinyarwanda sentence embedding model combining language-specific pretraining with curriculum training. It reports leading performance on Kinyarwanda STS and clustering, alongside public releases for research and deployment.

  • KinyaEmbed is presented as the first dedicated sentence embedding model for Kinyarwanda.
  • ρ = 0.7298 on SemRel2024-rw, with KinyaEmbed outperforming all baselines by at least 20.9%.
  • 8.6% is KinyaEmbed’s lead on Wiki-RW-STS, a fresh benchmark without training contamination.
  • KinyaEmbed achieves the best Silhouette Score across all evaluated models for document clustering.
  • The authors release the model, 2,936 filtered KinyaCOMET training pairs, and the Wiki-RW-STS benchmark publicly.

Ethical Statement

The paper states that its training and evaluation resources are publicly available, appropriately licensed, and free of personally identifiable information. It frames the technology as supporting Kinyarwanda-language accessibility without identified dual-use concerns.

  • All training data is described as publicly available and properly licensed, including the Gazette, OPUS-100, FLORES-200, KinyaCOMET, and Wikipedia.
  • The dataset and evaluation materials contain no personally identifiable information.
  • The paper describes the technology as enabling native-language information access for Kinyarwanda speakers without dual-use concerns.

A. Wiki-RW-STS Benchmark Construction

Wiki-RW-STS is constructed from Kinyarwanda Wikipedia using controlled sampling, native-speaker annotation, and agreement-based replacement of disputed pairs. The benchmark and related KinyaCOMET filtering process provide fresh evaluation data and quality-controlled semantic pairs.

  • Source corpus: ≈78,000 Kinyarwanda Wikipedia article paragraphs from the 20231101.rw snapshot form the source pool after excluding stubs and unsuitable articles.
  • Pair sampling: 300 pairs are sampled across high, medium, and low similarity conditions using within-paragraph, within-article, and lexical-similarity criteria.
  • Human annotation: Two native Kinyarwanda speakers score all pairs on a 0–1 relatedness scale, with final scores averaged across annotators.
  • Human annotation: Pairs with absolute annotator disagreement above 0.30 are discarded and replaced by resampling; 11 pairs are replaced.
  • Benchmark release: Wiki-RW-STS is released under CC-BY-SA 4.0.
  • KinyaCOMET filtering: KinyaCOMET pairs with quality score ≥0.8 are repurposed as semantic equivalents for MNRL training because lower-quality translations add noisy supervision.
  • KinyaCOMET filtering: 2,936 of 4,323 annotated KinyaCOMET pairs pass the threshold, spanning legal, religious, news, and government domains.
Loading 2608.26941v1…