PaperlayerView exact revision on arXiv

Source-linked AI summary

TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation

Milan Gritta, Patrik Lambert, Jihye Back, Amril Nazir

arXiv:2608.18655v1cs.CL

TL;DR

African machine translation lacks strong small models and sufficient high-quality open-source parallel data. TranslatePsy-AfriSLM addresses this with curated and synthetic data plus quality filtering, reducing training tokens by up to 96% while enabling 0.8B-parameter models to outperform much larger systems.

  • Problem

    African machine translation lacks performant small language models and sufficient high-quality open-source parallel data, while larger open-source models underperform and incur substantial running costs.

  • Method

    TranslatePsy-AfriSLM curates open-source and synthetic sentence pairs with unified multi-metric quality estimation, quantity selection, and bidirectional expansion for 19 languages.

  • Results

    Up to 96% of training tokens can be removed while maintaining comparable performance; filtered synthetic data dominates the quality-efficiency frontier and enables 0.8B-parameter models to outperform much larger systems.

  • Takeaways & Limitations

    Filtered synthetic generation appears to be the most practical path forward for African machine translation given the shortage of high-quality open-source parallel data.

  • Takeaways & Limitations

    Absolute translation quality remains uncertain, and reference-based metrics suggest African-language translation has not yet reached the quality of European and Asian languages.

Abstract

from arXiv · show

The rapid progress in Artificial Intelligence has largely bypassed African languages, creating a digital divide that limits AI adoption on the continent. Recent open-source LLMs systematically underperform on African machine translation, while the lack of large-scale, high-quality, open-source parallel data has constrained the development of competitive small language models (SLMs). We introduce *TranslatePsy-AfriSLM*, a collection of open-source MT resources for 19 Sub-Saharan African languages, including curated parallel data, African-specialized synthetic data, and a family of fine-tuned SLMs. Our empirical study shows that unified quality-estimation filtering removes up to 96% of training tokens without degrading quality, and that filtered synthetic data dominates the quality-efficiency Pareto frontier. Fine-tuned on the resulting data mixture, TranslatePsy-AfriSLM outperforms substantially larger systems, including TranslateGemma-27B and Qwen3.5-122B-A10B, with as few as 0.8B parameters.

1 Introduction

AI underinvestment has created major adoption barriers across Africa, especially for machine translation, where performant small language models and high-quality open-source data remain scarce. TranslatePsy-AfriSLM addresses this gap, with its smallest model outperforming dedicated NLLB models while retaining conversational capabilities.

  • Motivation: The lack of performant SLMs for African machine translation limits cross-border communication, trade, education, and broader AI adoption for over a billion people (Ssemugabi, 2025; Moukatib and SeddThe introduction attributes this barrier to AI underinvestment on the continent (Nwagbala et al., 2025; Diallo et al., 2025; Isangula, 2025) and its effects on productivity and collaboration.
  • Problem: Large MT systems can achieve stronger African translation performance than existing alternatives but incur substantial running costs because of their large parameter counts.The passage identifies large parameter counts as the source of the substantial running-cost burden.
  • Data challenge: High-quality open-source data is too scarce for efficient SLM adaptation, while internet repositories are noisy and unstructured, training signals are sparse, and token budgets are inflated.Existing curated datasets and human-quality translations are too small to provide more than modest improvements.
  • Results: The smallest TranslatePsy-AfriSLM model outperforms dedicated NLLB encoder-decoder models while preserving conversational capabilities.The broader comparison is illustrated by Figure 1, where TranslatePsy-AfriSLM surpasses Qwen3.5-122B-A10B and TranslateGemma-27B on SSA-COMET scores on the BOUQuET benchmark.

2 Background

Existing African MT resources trade off scale, language coverage, and translation quality across large unstructured repositories, medium curated corpora, and small human-quality datasets. African-centric quality-estimation metrics have been used for filtering, but their comparative usefulness and broader filtering questions remain unresolved.

  • Dataset Landscape: African MT datasets fall into large unstructured, medium-sized curated, and small human-quality categories, reflecting a persistent scale–coverage–quality trade-off.Large repositories provide quantity but suffer from arbitrary sizes, duplication, contamination, and noise; curated and human-quality datasets offer stronger quality but narrower coverage or limited quantities.
  • Large but Unstructured: Large repositories such as OPUS, MALA (Ji et al., 2024), WMT22 (Adelani et al., 2022b), and Fine Translations offer extensive parallel data but contain duplication, contamination, and noisy text.NLLB (Team et al., 2022) leveraged approximately 18B sentence pairs across 200 languages, illustrating the scale available in this category.
  • Medium-Sized and Curated: AfriNLLB2 is the only readily available medium-sized curated dataset, but it covers 9 African languages and about 50% of its approximately 3M pairs include Arabic or European languages.AfriqueLLM (Yu et al., 2026) improves African-language MT through continued pretraining, but its training data was not publicly released.
  • Small but Human-Quality: Small human-quality datasets provide strong translations but limited quantities and uneven language coverage, yielding only modest improvements that are smaller than AfriNLLB and substantially weaker than larger alternatives.Examples include MMT-Africa (Emezue and Dossou, 2021), AfriDOC-MT (Alabi et al., 2025), SMOL (Caswell et al., 2025), LAFAND-MT (Adelani et al., 2022a), WMT24pp (Deutsch et al., 2025), and MENYO-MT (Adelani et al., 2021).
  • Quality Estimation: Afri-COMET (Wang et al., 2024) and SSA-COMET are African-centric reference-free metrics used to filter African parallel text, but their comparative usefulness as training-data filters remains unanswered.Prior filtering work includes Yu et al. (2026), Uemura et al. (2026), and Moslem et al. (2026), alongside COMET (Rei et al., 2020), COMETKIWI (Rei et al., 2022b), and MetricX.

3 TranslatePsy-AfriSLM

TranslatePsy-AfriSLM builds a multi-step curation pipeline for heterogeneous African MT data, combining preprocessing, unified quality estimation, and quantity-selection strategies. It supplements open-source and synthetic data with human, instruction-following, and broader multilingual mixtures.

  • Data curation: The pipeline applies structural preprocessing, Unified Quality Estimation, and post-training data-quantity selection to open-source and synthetic sentence pairs.Preprocessing includes cleaning, language checks, deduplication, and test-set decontamination.
  • Data sources: Open-source data comes from WMT22, MALA, OPUS, and Fine Translations, while synthetic data translates MADLAD-400 monolingual data with the NLLB-3.3B teacher.All available data was processed for 19 African languages, while 3.6 million English documents were randomly sampled.
  • Quality estimation: AfriCOMET, SSA-COMET, and MetricX scores are normalized into a shared robust z-score calibrated on approximately 352K human-translated pairs.Polarity correction makes higher scores consistently indicate higher quality, enabling aggregation across metrics, directions, and data sources.
  • Quality estimation: Filtering compares aligned, reversed, and mean directional scoring because quality-estimation scores can differ when a pair’s argument order is reversed.The mean strategy averages both directional z-scores, while aligned and reversed strategies score pairs in the training or opposite direction.
  • Data selection: Data selection uses threshold filtering, TopN caps per-language-pair examples, and bidirectional expansion that reverses sentence pairs to balance translation directions.TopN mitigates language-distribution imbalances, while bidirectional expansion is an orthogonal augmentation step.

4 Experimental Setup

The experiments fine-tune Qwen3.5 models with full-parameter SFT on a one-epoch final training mixture, evaluating translation across 19 covered and 8 unseen African languages. Evaluation spans three benchmarks, four complementary metrics, and comparisons with general-purpose, dedicated, and African-specialized models.

  • Model training: Qwen3.5 models undergo full-parameter supervised fine-tuning for one epoch on the final training mixture, computing loss only on assistant tokens.Qwen3.5 was selected because preliminary benchmarks found it consistently outperformed other general-purpose SLMs.
  • Language groups: Evaluation covers 19 TranslatePsy-AfriSLM languages in Africa-IID and 8 additional unseen languages in Africa-OOD to assess out-of-domain generalisation.The Africa-IID and Africa-OOD groups are defined as the covered and unseen language sets, respectively.
  • Benchmarks: The study evaluates Flores-200, BOUQuET, and Smol, combining a widely used benchmark with newer linguist-curated and professionally translated test sets.Flores-200 uses 1,012 devtest sentences, BOUQuET 854 test sentences, and Smol 863 smolsent sentences.
  • Metrics: Performance is measured with COMET-22, SSA-COMET, MetricX, and ChrF++, with higher-is-better scales for the first, second, and fourth metrics but lower-is-better MetricX.COMET-22 and SSA-COMET use 0–1 scales, ChrF++ uses 0–100, and MetricX uses 0–25.
  • Baselines: Baselines include Qwen3, Qwen3.5, Apertus, NLLB, AfriNLLB, TranslateGemma, Hunyuan-MT, and AfriqueLLM, spanning general-purpose, dedicated, and African-specialized models.Qwen3.5 is reported as the strongest general-purpose baseline and is therefore used as the backbone.

5 Results and Analysis

TranslatePsy-AfriSLM shows that aggressive quality filtering and high-quality synthetic data enable efficient African MT adaptation, with a 32.37B-token final mix reaching 0.632 SSA-COMET. The resulting 0.8B–4B SLMs outperform larger general-purpose and dedicated translation models, transfer to unseen African languages, and preserve non-African translation ability when supplemented with broader multilingual data.

  • Data quality and quantity: A filtered configuration matches the unfiltered model’s SSA-COMET score, 0.530 vs. 0.528, using only 1.76B rather than 44.93B tokens—a 96% reduction.The result indicates that curation concentrates the useful training signal in raw open-source data.
  • Data quality and quantity: Synthetic data supports stricter quality thresholds than open-source mixes, while combined mixes help mainly at smaller budgets and can underperform synthetic-only mixes at larger scales.Open-source mixes typically use thresholds around z̄ ∈[−0.5, 0.5], whereas synthetic mixes remain viable from z̄ ≥0.68 to higher thresholds; the best large combined mix uses 46.49B tokens at z̄ ≥0.41 versus 32.37B tokens at z̄ ≥0.68 for the final synthetic mix.
  • Data quality and quantity: The final synthetic mix combines a threshold with bidirectional expansion, retains 32.37B tokens, and reaches 0.632 SSA-COMET.Bidirectional expansion improves every tested open-source, synthetic, and combined configuration, while the selected synthetic mix becomes the final training dataset.
  • Data sources: TranslatePsy-AfriSLM data combines scale and quality more effectively than prior sources, while human-quality data is valuable per token but too limited in scale for standalone SLM post-training.Human Mix contains 60.2M tokens versus 535M for AfriNLLB, and Instruct-Mix unexpectedly improves MT, likely because its 2.3M examples cover diverse African-language tasks.
  • Africa-IID: The 0.8B model surpasses Qwen3.5-122B-A10B on Flores-200, BOUQuET, and Smol, while TranslatePsy-AfriSLM-0.8B also outperforms AfriNLLB-600M, NLLB-3.3B, and other dedicated translation models.Paired bootstrap tests confirm significant gains over larger LLM baselines on most settings, and TranslatePsy-AfriSLM-2B consistently surpasses dedicated NLLB models.
  • Africa-OOD and catastrophic forgetting: TranslatePsy-AfriSLM-2B improves over Qwen3.5-2B on all eight held-out Africa-OOD languages, while Asia-Europe Mix reduces MetricX degradation from −86.0% to −10.3%.The transfer gains are especially large for Sepedi, Bambara, and Akan, and broader multilingual data also mitigates degradation on Asian and European languages.

6 Conclusions

TranslatePsy-AfriSLM shows that rigorous quality-estimation filtering can sharply reduce training data while preserving translation performance for 19 Sub-Saharan African languages. Filtered synthetic data offers the strongest quality-efficiency tradeoff and enables a 0.8B-parameter model to outperform much larger systems.

  • 6 Conclusions: Up to 96% of training tokens can be removed while maintaining comparable performance by combining normalized QE metrics and scoring examples in the training direction.Robust ¯z score normalization makes filtering more consistent across multiple quality-estimation metrics.
  • 6 Conclusions: With as few as 0.8B parameters, TranslatePsy-AfriSLM outperformed much larger models, including TranslateGemma-27B and Qwen3.5-122B-A10B.The model was post-trained on the highest-quality data identified by the study.

Limitations

Absolute translation quality for African languages remains difficult to determine, motivating human evaluation despite reference-free estimators indicating that TranslatePsy-AfriSLM data is comparable to human-translated pairs at much larger scale.

  • The Need for Human Evaluation: Human evaluation is needed to determine absolute translation quality for African languages, because reference-free estimators only suggest that TranslatePsy-AfriSLM data matches human-translated pair quality.The data is available in quantities orders of magnitude larger than human-translated pairs, but the passage indicates that absolute quality remains unresolved.

Ethics Statement … A.5 AfriNLLB

The appendix documents the data construction, filtering, composition, and limitations of TranslatePsy-AfriSLM, including licensing, representativeness, synthetic-content concerns, and preservation of non-African capabilities.

  • Ethics Statement: All primary sources use permissive attribution-based or research-permissive licenses, and the compiled datasets will be released similarly for reproducible low-resource NLP research.Sources include ODC-By v1.0, CC BY 4.0, OPUS, and WMT22.
  • Ethics Statement: The corpus may over-represent standardized written forms, under-represent regional dialects and oral traditions, and include partially synthetic content from recycling, duplication, correction, and paraphrasing.The authors recommend auditing trained models for localized sensitivity before deployment.
  • A.1 Synthetic and Open-Source Mixes: Synthetic and open-source mixes undergo document and sentence processing, language identification, cleaning, filtering, exact and fuzzy deduplication, and quality-estimation filtering before bidirectional expansion.NeMo Curator implements preprocessing through decontamination; splitting applies only to monolingual source texts used for synthetic generation.
  • A.2 Human Mix: The human mix contains 352,582 human-translated high-quality parallel sentences from AfriDOC-MT and SMOL18, expanded bidirectionally and deduplicated with evaluation-set decontamination.The SmolDoc subset of SMOL19 was flattened to sentence-level pairs, followed by global exact and approximate deduplication.
  • A.3 Instruct Mix: The Afri-Instruct mix combines 2,306,800 examples from 11 public datasets, led by open-ended instruction and chat data at 1,553,944 examples (67.36%) and code assistance at 521,389 (22.60%).Datasets were standardized into multi-turn chat format, filtered for empty messages, expanded for AfriQA, and sampled across tasks.
  • A.3 Instruct Mix: The General-Instruct mix adds public smoltalk2 and Dolci-Instruct examples with unified chat formatting and exact deduplication, but excluding reasoning data removes the base model’s thinking capability.Exact deduplication removed 85,982 examples.
  • A.4 Asia-Europe Mix: To mitigate catastrophic forgetting, the Asia-Europe mix includes 38 medium-high-resource languages and 24,114,303 en→xx parallel sentences, preserving Asian and European performance with negligible African-quality impact.The data is quality-filtered using mean bidirectional COMET ≥0.6, then deduplicated and decontaminated.
  • A.5 AfriNLLB: AfriNLLB preprocessing applies test-data decontamination and approximate MinHash deduplication using character 4-grams, with details summarized in Table 13.The MinHash configuration uses 128 permutations and a Jaccard similarity of 0.8.

A.6 Data Decontamination · B Additional Experimental Setup · B.1 Translation Prompt

The paper decontaminates training data against more than 850K evaluation sentences using granular MinHash filtering, then structures translation examples with language and text fields before applying model-specific chat templates.

  • A.6 Data Decontamination: Over 850K evaluation sentences, languages, and dataset splits were used to decontaminate the training data.The procedure covers all evaluation sentences, languages, and dataset splits.
  • A.6 Data Decontamination: BPE-unigram Qwen3.5 MinHash with locality-sensitive hashing filtered training data at a Jaccard similarity threshold of 0.9.This deduplication configuration was part of the preprocessing pipeline.
  • A.6 Data Decontamination: Deduplication operated at the individual-sentence level rather than the pair level for more granular detection.The sentence-level design was selected to maximize detection granularity.
  • B.1 Translation Prompt: Training and evaluation examples were structured using SOURCE_LANG, TARGET_LANG, SOURCE_TEXT, and TARGET_TEXT fields.The language fields identify source and target languages, while the text fields represent input and output text.
  • B Additional Experimental Setup: A model-specific chat template was applied after the training and evaluation examples were structured.This was the final prompt-formatting step described in the setup.
  • B.1 Translation Prompt: Figure 8 presents the training and evaluation prompt template used for these structured examples.The figure documents the prompt format for both training and evaluation.

B.2 Training Hyperparameters

Training uses full fine-tuning with a small peak learning rate, AdamW optimization, and a global batch size of 256. Samples are formatted as translation prompts, length-filtered, packed, and trained on bfloat16 H100 infrastructure.

  • Optimization: Peak learning rate is 1.25 × 10−5, using fused AdamW, a linear schedule with 1% warmup, gradient clipping at 1.0, checkpointing, and global batch size 256.These settings define the full fine-tuning optimization procedure.
  • Data formatting: Training samples use the translation prompt format and model-specific chat templates, while sequences over 2,048 tokens are filtered and the rest packed via best-fit decreasing.Formatting and packing are applied before training.
  • Infrastructure: Experiments run with PyTorch, HuggingFace Transformers, TRL, and DeepSpeed ZeRO-2 in bfloat16 on 32 NVIDIA H100 GPUs.The infrastructure configuration is shared across all experiments.

C Catastrophic Forgetting Analysis · D Additional Results

The Asia-Europe Mix mitigates catastrophic forgetting on non-African languages while preserving Africa-IID gains. It is retained in the final TranslatePsy-AfriSLM recipe to preserve broader multilingual translation.

  • C Catastrophic Forgetting Analysis: The analysis tests whether the Asia-Europe Mix preserves non-African translation performance without compromising African MT.
  • C Catastrophic Forgetting Analysis: The mix adds bidirectional English parallel data covering Asian and European language groups.
  • C Catastrophic Forgetting Analysis: Without the mix, fine-tuning substantially degrades Asian and European language-pair performance, revealing catastrophic forgetting beyond the African training distribution.
  • C Catastrophic Forgetting Analysis: Adding the mix consistently reduces degradation across all metrics and both non-African language groups.
  • C Catastrophic Forgetting Analysis: −95.1% to −16.8%: the MetricX-24 drop for Asian languages is reduced by adding the Asia-Europe Mix.
  • C Catastrophic Forgetting Analysis: −81.0% to −6.7%: the MetricX-24 drop for European languages is reduced by adding the Asia-Europe Mix.
  • C Catastrophic Forgetting Analysis: The retention benefit does not reduce Africa-IID performance, with nearly identical gains across metrics for models with and without the mix.
  • C Catastrophic Forgetting Analysis: The final recipe retains the Asia-Europe Mix to preserve broader multilingual translation while maintaining Africa-IID gains, rather than to improve African MT directly.

D.1 Per-Language Performance Analysis · D.2 Statistical Significance (Bootstrap)

TranslatePsy-AfriSLM improves performance across 19 Africa-IID languages and transfers to held-out Africa-OOD languages, with bootstrap tests showing stable benchmark rankings. The 0.8B model outperforms larger systems on several metrics while remaining competitive or mixed against NLLB.

  • D.1 Per-Language Performance Analysis: The Asia-Europe Mix substantially reduces degradation on Asian and European language pairs while preserving nearly identical gains on Africa-IID pairs.Figure 9 reports percentage changes from the Qwen3.5-2B baseline after supervised fine-tuning.
  • D.2 Statistical Significance (Bootstrap): Paired bootstrap tests use 10,000 sentence-pair resamples to estimate mean score differences and improvement-direction p-values across model pairs, datasets, and metrics.MetricX treats lower scores as better, whereas all other metrics treat higher scores as better.
  • D.1 Per-Language Performance Analysis: TranslatePsy-AfriSLM-2B consistently improves over Qwen3.5-2B across all 19 Africa-IID languages and four metrics, with largest gains for Oromo, Malagasy, Lingala, Tswana, and Zulu.Afrikaans shows smaller gains because its baseline is stronger.
  • D.1 Per-Language Performance Analysis: Africa-IID and Africa-OOD per-language scores average both translation directions where available, with OOD coverage varying across Flores-200, BOUQuET, and Smol.This evaluation design accounts for differing language subsets across the available datasets.
  • D.1 Per-Language Performance Analysis: TranslatePsy-AfriSLM-2B improves overall performance on Africa-OOD languages held out from fine-tuning, indicating transfer beyond languages seen during post-training.Scores average both directions and available Flores-200, BOUQuET, and Smol evaluation sets; strongest gains appear for Sepe, while some metric-specific regressions affect Nigerian Pidgin, Sudanese Arabic, and Tamazight.
  • D.2 Statistical Significance (Bootstrap): TranslatePsy-AfriSLM-0.8B substantially outperforms Qwen3.5-122B on SSA-COMET, MetricX, and ChrF++, remains competitive on COMET-22, and surpasses TranslateGemma-27B and AfriGemma-12B across reported settings.Against NLLB, the 0.8B model is competitive but mixed, while the 2B model significantly outperforms NLLB-1.3B and NLLB.
Loading 2608.18655v1…