PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

Mullosharaf K. Arabov

arXiv:2609.29559v1cs.CL

TL;DR

Arabic–Russian MT remains under-researched amid rich Arabic morphology and low lexical overlap. The paper benchmarks fine-tuned NMT against few-shot LLMs on a large parallel corpus, finding a consistent performance advantage for fine-tuned NMT.

  • Problem

    Arabic–Russian MT remains under-researched, with rich Arabic morphology and low lexical overlap between the typologically distant languages posing central challenges.

    Source: §1 · p. 1

  • Method

    The study benchmarks seven fine-tuned NMT models against four instruction-tuned LLMs using zero-shot and few-shot settings on a 20k-training-pair split.

    Source: §1 · p. 1

  • Results

    Fine-tuned NMT consistently outperforms few-shot LLMs; NLLB-1.3B scores BLEU 16.3 versus Aya-Expanse 8B’s 5.6 and COMET 0.738 versus 0.612.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Under the common low-resource scenario with only a few thousand parallel training sentences, dedicated NMT models remain preferable for Arabic–Russian translation.

    Source: §5 · p. 11

    Fine-tuning was limited to 20k sentence pairs by computational constraints, and larger training sets could improve quality and alter some model rankings.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Arabic-Russian machine translation (MT) remains under-explored due to the rich morphology of Arabic and low lexical overlap between the two languages. We benchmark seven fine-tuned neural machine translation (NMT) models against four few-shot large language models (LLMs) on a 20k/5k/5k split of a new 15.47M-pair corpus. Fine-tuned NLLB-1.3B achieves the highest BLEU (16.3) and COMET (0.738). Aya-Expanse 8B leads the few-shot LLMs (BLEU 1.7 on 500 sentences, chrF 25.7), but all LLM scores remain far below the fine-tuned NMT baselines. Error analysis identifies low lexical overlap as the dominant failure mode; among the worst translations, mT5-small produces 32% too-short outputs. Bootstrap tests confirm significant differences among most models. Our results demonstrate that fine-tuned NMT significantly outperforms few-shot LLMs for Arabic-Russian translation under low-resource conditions.

1 Introduction

Arabic–Russian MT is under-researched despite its practical importance, with rich Arabic morphology and low lexical overlap posing central challenges. This work benchmarks fine-tuned NMT against few-shot LLMs and finds the fine-tuned systems ahead.

1 Introduction

  • Arabic–Russian MT faces two central challenges: rich Arabic morphology and low lexical overlap between the typologically distant languages.Arabic morphology includes a non-concatenative root-and-pattern system, complex cliticization, and substantial dialectal variation.Source: §1 · p. 1
  • The study compares seven fine-tuned NMT models with four instruction-tuned LLMs using zero-shot and few-shot evaluation under a 20k-training-pair setting.The best LLM is evaluated on 5,000 sentences after initial 500-sentence comparisons.Source: §1 · p. 1
  • NLLB-1.3B leads the fine-tuned NMT models with BLEU 16.3 and COMET 0.738, while few-shot LLMs remain substantially below the best fine-tuned system.The benchmark uses a 20k/5k/5k split.Source: §1 · p. 2
  • The work contributes a benchmark, bootstrap-based statistical evaluation, an error taxonomy, and an open-source reproduction toolkit.It does not propose a new model architecture.Source: §1 · p. 2

2 Related Work

Prior work has addressed Arabic preprocessing, parallel resources, parameter-efficient adaptation, and transliteration, while systematic Arabic–Russian benchmarking across fine-tuned NMT and few-shot LLMs remained unreported.

Arabic preprocessing and morphology

  • Arabic preprocessing research includes joint tokenization, part-of-speech tagging, and morphological disambiguation, alongside MADAMIRA and Farasa tools [Habash and Rambow, 2005; Pasha et al., 2014; Darwish and Mubarak, 2016].The cited survey reviews subword tokenization and contextual representations [Alrekabee, 2025].Source: §2 · p. 2

Arabic corpora and parallel resources

  • The study draws on the Arabic–Russian Translation Corpus and its curated six-domain subset, while MADAR provides a resource for dialectal variation.The curated subset includes religion, dictionaries, the Bible, Tatoeba, news, and conversation.Source: §2 · p. 2

Previous Arabic–Russian MT work

  • Earlier Arabic–Russian work used a 27k-pair scientific-domain corpus and three LoRA/QLoRA-tuned multilingual models; this study broadens the comparison to a larger general-domain corpus and more models [Arabov, 2026a].The earlier work reported that fine-tuning outperformed few-shot prompting and that scientific translation needed domain-specific fine-tuning.Source: §2 · p. 2

Parameter-efficient fine-tuning

  • For low-resource adaptation, prior Bashkir work found QLoRA on 7B-scale models offered a quality–cost trade-off, and this study adopts LoRA and QLoRA [Arabov and Khaybullina, 2026].The cited work concerns Bashkir, a low-resource agglutinative language.Source: §2 · p. 2

Transliteration as a related task

  • Related transliteration research found byte-level models outperforming subword multilingual models for Tajik–Farsi, a distinct task that also involves script and phonological divergence [Arabov, 2026b].The passage relates this divergence to Arabic–Russian MT without treating transliteration as translation.Source: §2 · p. 2

Previous Arabic–Russian MT work

  • This study identifies a gap in systematic benchmarking of fine-tuned NMT and few-shot LLMs on a large general-domain Arabic–Russian corpus.The authors characterize the focus as rich morphology and low lexical overlap.Source: §2 · pp. 2–3

3 Experimental Setup

The experiments sample a low-resource training set from a large filtered Arabic–Russian corpus, compare fine-tuned NMT and prompted LLMs, and assess translation quality, resource costs, and common errors.

Dataset

  • 15,467,945 filtered sentence pairs form the corpus, with OPUS as its largest source; the study samples 20k/5k/5k pairs for training, validation, and test.Manual inspection found 96% alignment quality, and the split uses seed 42.Source: §3 · p. 3, Table 1 · p. 3

Zero-shot and Few-shot LLM Prompting

  • Four instruction-tuned LLMs are tested zero-shot and with five examples on 500 sentences, then the best model is evaluated on 5,000 sentences.The prompting design uses a fixed seed.Source: §3 · p. 3

Evaluation Metrics and Statistical Analysis

  • Evaluation reports BLEU, chrF, BERTScore, and COMET, with 95% bootstrap confidence intervals and pairwise significance tests using 1,000 resamples.NMT uses the full 5,000-sentence test set; initial LLM comparisons use 500 sentences.Source: §3 · p. 3

Fine-tuning of NMT Models

  • Seven NMT architectures are fine-tuned for three epochs on 20k pairs using full fine-tuning, LoRA, or QLoRA, with the best checkpoint selected by validation sacreBLEU.The adaptation methods and hyperparameters vary by model family.Source: §3 · p. 3

Computational Cost

  • Table 2 reports training time and peak GPU memory; QLoRA models take the longest to train but use less memory than full-parameter counterparts.Experiments ran on a single NVIDIA L4 GPU with 24 GB of VRAM.Source: §3 · p. 3, Table 2 · p. 4

Error Classification

  • The error taxonomy classifies each model’s 50 lowest-scoring translations using categories including too-short, too-long, repetition, and low-overlap outputs.Thresholds derive from the training-data length distribution and heuristics.Source: §3 · p. 4

4 Results

On a shared 500-sentence subset, fine-tuned NMT significantly outperforms few-shot LLMs; NLLB-1.3B leads the fine-tuned models, while Aya-Expanse 8B leads the LLMs. Results also show sentence-length sensitivity, low lexical overlap as a dominant error, correlations among metrics, and distinct model behaviors in examples.

4.1 Main Comparison

  • NLLB-1.3B leads fine-tuned NMT with BLEU 16.30, chrF 34.98, TER 90.83, COMET 0.738, and BERTScore 0.783.Within mT5, performance scales with model size: mT5-small nearly fails, while mT5-large remains behind specialized NMT architectures.Source: Table 3 · p. 4, §4.1 · p. 4, Table 4 · p. 5
  • In zero-shot evaluation, Aya-Expanse leads BLEU while Llama 3.1 leads chrF, BERTScore, and COMET; five examples improve all LLMs and make Aya-Expanse the few-shot leader.Aya-Expanse’s few-shot BLEU is 1.69 and chrF is 25.75; its chrF rises 6.31 points from zero-shot.Source: §4.1 · pp. 4–5, Table 5 · p. 5, Table 6 · p. 5

Direct NMT vs. LLM comparison on the shared 500-sentence subset

4.2 Statistical Significance

  • Paired tests confirm significant differences among leading fine-tuned NMT models and the ranking Aya-Expanse > Llama 3.1 > Qwen 2.5 > Mistral among few-shot LLMs.NLLB-1.3B significantly outperforms Aya-Expanse across all four metrics, and mT5-large exceeds every LLM on BLEU, chrF, and COMET.Source: §4.2 · pp. 7–8

4.3 Impact of Source Sentence Length

  • All models degrade on the longest 20% of sentences, but Aya-Expanse loses about 4 BLEU points from Q3 to Q5 versus NLLB-1.3B’s 1.5-point loss.NLLB-1.3B remains relatively stable across length quantiles, while few-shot LLM BLEU drops sharply beyond 120 characters.Source: §4.3 · p. 8

4.4 Error Analysis

  • Low lexical overlap accounts for 84–96% of worst-case errors; mT5-small produces too-short outputs in 32% of its worst translations and 48% of the full test set.Few-shot LLMs also often select fluent synonyms or paraphrases absent from the reference; excessive length is most notable in Marian (5.6%) and M2M100 (4.3%).Source: §4.4 · pp. 8–9, Table 13 · p. 8, Table 14 · p. 8

4.5 Correlation Between Metrics

  • Among fine-tuned models, chrF and BERTScore correlate most strongly (ρ=0.855), while BLEU’s moderate correlations indicate it captures a somewhat different aspect of quality.TER correlates negatively with the other metrics, and the authors use these relationships to justify multiple complementary metrics.Source: §4.5 · p. 9, Table 15 · p. 9

4.6 Qualitative Examples

  • The examples show mT5-small collapsing, Marian and NLLB-1.3B omitting content, mT5-base shifting meaning, and Aya-Expanse capturing an idiom but adding redundant context.In the second example, Aya-Expanse is fluent and close to the reference, while NLLB-1.3B truncates its output.Source: §4.6 · p. 9

5 Discussion and Conclusion

Fine-tuned NMT consistently outperforms few-shot LLMs for Arabic–Russian translation, with NLLB-1.3B leading the benchmark. The discussion also highlights practical training-cost trade-offs, error patterns, and limits on evaluation evidence.

5 Discussion and Conclusion

  • NLLB-1.3B scores BLEU 16.3, COMET 0.738, and BERTScore 0.783, versus Aya-Expanse 8B’s 5.6, 0.612, and 0.692, respectively.On the shared 500-sentence subset, COMET is 0.746 versus 0.654 (p<0.001).Source: §5 · p. 9
  • Aya-Expanse 8B gains +6.31 chrF and +0.042 COMET from zero-shot to few-shot, compared with Llama 3.1’s +0.93 chrF and +0.015 COMET.The discussion attributes the pattern to translation-oriented instruction tuning enabling more effective use of in-context examples, and reports mT5 scaling from near failure to performance that still trails specialised NMT.Source: §5 · p. 9
  • Marian reaches BLEU 15.48 in 10 minutes, while NLLB-1.3B with QLoRA reaches BLEU 16.30 after over six hours.The paper recommends Marian for rapid prototyping or resource-constrained environments, and NLLB when small quality improvements matter.Source: §5 · p. 10
  • mT5-base’s BLEU 6.54 exceeds mT5-small’s 1.51, but their TER scores are comparable at 106.7 and 103.7, respectively.The authors suggest mT5-base produces longer but still largely incorrect translations, so BLEU alone may overestimate weak models’ practical usability.Source: §5 · p. 10
  • Low lexical overlap accounts for 84–96% of the worst fine-tuned NMT translations, while LLMs often produce fluent Russian with reference-mismatching synonyms or paraphrases.LLMs also lose up to 4 BLEU points from the third to fifth source-length quantile; chrF, BERTScore, and COMET correlate at ρ≥0.81.Source: §5 · p. 10
  • NLLB-1.3B scores BLEU 16.40 on the 500-sentence subset and 16.30 on the full test set, supporting the representativeness of subset comparisons within bootstrap confidence intervals.The head-to-head LLM comparisons were limited to 500 sentences.Source: §5 · p. 11
  • The study did not conduct large-scale human evaluation, though the authors consider the evidence robust because results are stable across automatic metrics and bootstrap testing.The passage notes COMET’s reported correlation above 0.9 with human judgments in WMT shared tasks [Rei et al., 2020].Source: §5 · p. 11
  • Across the 20k-pair low-resource setting, the study concludes that dedicated NMT models remain preferable to general-purpose LLMs for Arabic–Russian translation.It presents the benchmark as a baseline for the common low-resource scenario in which only a few thousand parallel sentences are available for training.Source: §5 · p. 11

Limitations

The study’s findings are bounded by its training-set size, limited full-scale LLM evaluation, single-reference scoring, corpus coverage, and heuristic error thresholds.

Limitations

  • All models were fine-tuned on only 20k sentence pairs due to computational constraints; larger sets may improve absolute quality and alter some model rankings.The authors describe 20k pairs as realistic for low-resource settings.Source: Limitations · p. 11
  • Only the best-performing LLM was evaluated on the full 5,000-sentence test set, leaving full-scale evaluation of all four LLMs for future work.The available full-scale result for Aya-Expanse confirms that the observed gap persists.Source: Limitations · p. 11
  • All automatic metrics use one Russian reference per source, which can penalise valid lexical variation and overestimate low-overlap errors.Source: Limitations · p. 11
  • The corpus is predominantly Modern Standard Arabic from OPUS, limiting generalisability to dialectal Arabic and specialised domains.The authors describe this distribution as representative of currently available Arabic–Russian parallel data.Source: Limitations · p. 11
  • Error-category thresholds were chosen heuristically from training-data length distributions and may need adjustment for other language pairs or domains.Source: Limitations · p. 11

6 Future Work

Future work targets broader training resources, methods to address lexical overlap, expanded model and dialect coverage, better-tailored evaluation, and human validation.

6 Future Work

  • The authors plan to add domain-specific dictionaries for medicine, engineering, law, and military terminology, and fine-tune on the full 15.8M-pair corpus.The larger-corpus experiments are intended to establish an upper-bound performance for the language pair.Source: §6 · p. 11
  • Explicit word alignment and transliteration-based preprocessing may help reduce the lexical-overlap bottleneck identified in the error analysis.Source: §6 · p. 11
  • Planned evaluations include newer multilingual and retrieval-augmented models, plus dialectal Arabic using resources such as MADAR.Source: §6 · p. 11
  • The authors aim to develop morphology-aware evaluation metrics tailored to Arabic–Russian, building on Arabic preprocessing and representation surveys.Source: §6 · p. 11
  • The authors plan targeted human evaluation on samples of low-overlap and morphologically complex sentences.Source: §6 · p. 11
Loading 2609.29559v1…