ReMova: Fine-tuning LLMs for English to Belarusian translation

1 Introduction

• Belarusian remains under-prioritised in machine-translation training and evaluation because web languages and linguistic resources are unevenly distributed.The paper presents a Belarusian-specific cleaning pipeline and fine-tunes models from encoder-decoder and decoder-only paradigms.Source: §1 · p. 1

1.1 Belarusian variation

• Belarusian has competing normative systems, and interference from other languages can affect text at lexical, phonetic, morphological, syntactic, and stylistic levels.The paper focuses on standard Belarusian orthography; extending models to other varieties requires dedicated evaluation and training data.Source: §1.1 · pp. 1–2
• Prior research reports that machine-translated forms entered media and ordinary Belarusian usage, raising questions about the representativeness of some textual data.That research describes artificial lexical items, grammatical constructions, and semantic shifts.Source: §1.1 · p. 2

1.2 Importance of data cleaning

• The proposed cleaning approach stabilises lexical, orthographical, and typographical features under standard Belarusian orthography while accepting new word-forms and leaving syntax for future work.It aims to distinguish normative systems from obvious errors and language interference without imposing views on language change.Source: §1.2 · p. 2

2 Methodology

• REMOVA combines general-purpose text processing and language identification with Belarusian-specific tools and quality metrics to extract reliable parallel segments.The pipeline stages write outputs and record summary statistics; filtering applies thresholds to accumulated metrics, and pairs of two words or fewer are discarded beforehand.Source: §2 · p. 2, §2.1 · p. 3

2.1 Belarusian-specific normalisation

• Wikificator targets common orthographic errors and Taraškievica variants, while LanguageTool provides broader dictionary-based spelling and grammar coverage.About 12% of pairs were dropped at this stage, mostly for exceeding a manually selected LanguageTool-density threshold or being flagged as duplicates; the dictionary was iteratively curated to address false positives.Source: §2.1 · p. 3

2.2 Training data

• The training data combine five sources with corpus-specific BLASER cutoffs calibrated to register, since a uniform cutoff would discard much of the subtitle-style signal.Sources include web-crawled bitext, subtitles, Tatoeba, Wikimedia, and back-translated data; selected informal HPLT material adds 4,600 back-translated pairs, with 84% textually distinct targets.Source: §2.2 · pp. 3–4

3.1 Identifying strongest baseline

• Base models were screened to identify a strong Belarusian translation baseline, with an initial under-4B-parameter comparison conducted without fine-tuning.The study considered encoder-decoder systems and decoder-only LLMs, including suggested multilingual models and Llama-3.2.Source: §3.1 · p. 4

3.2 Fine-tuning

• The selected models use full fine-tuning for NLLB and LoRA-based supervised fine-tuning for Gemma-4-12B-IT and TranslateGemma-12B-IT.The training corpus mixes sentence pairs with paragraph-level and multi-sentence alignments; evaluation uses BOUQuET at native sentence granularity.Source: §3.2 · p. 4

3.3 Filtering ablation

• The filtering ablation re-fine-tunes the selected models on unfiltered data from the same sources, matching cleaned-run volume, source proportions, hyperparameters, and evaluation protocol.This design is intended to separate filtering effects from differences in data quantity or composition.Source: §3.3 · pp. 4–5

3.4 Evaluation

• Evaluation uses the held-out test split of BOUQuET, containing 854 sentences across eight domains, and reports BLEU, chrF++, and COMET.The dev split selects checkpoints; base LLMs use an identical fixed three-shot prompt, and chrF++ is noted as more informative for morphologically rich languages.Source: §3.4 · p. 5

4.1 Base models

• NLLB-200 scores 27.2 BLEU versus MADLAD-400’s 21.8, while Gemma-3 reaches 22.9 BLEU at 4B among the tested base models.Only NLLB-200 and the Gemma family cleared the bar for productive fine-tuning.Source: §4.1 · p. 5

4.2 Fine-tuned models

• TranslateGemma leads aggregate results with 32.63 BLEU and 55.36 chrF++; Gemma-4 scores 31.40 BLEU, while NLLB slightly exceeds Gemma-4 on chrF++.Gemma-4 with LoRA on cleaned data reaches parity with fully fine-tuned NLLB.Source: §4.2 · pp. 5–6, Table 5 · p. 6
• TranslateGemma fine-tuning adds +7.65 BLEU and +3.63 chrF++, with BLEU improving in every domain but by uneven amounts.Gains range from +9.55 BLEU on Narration and +9.12 on Conversation to +3.61 on Other misc.; chrF++ falls by 0.17 only in Other misc.Source: §4.2 · p. 6
• Narration BLEU gains span +5.2 for NLLB, +9.1 for Gemma-4, and +9.6 for TranslateGemma; the authors plausibly link this pattern to OpenSubtitles’ 35% data share and 2× upsampling.The domains with the smallest gains differ by model; diversifying training styles is identified as future work.Source: §4.2 · p. 6

4.3 Filtering ablation

• Cleaning adds +1.99 BLEU / +1.10 chrF++ for NLLB, +3.62 / +1.53 for Gemma-4, and +4.38 / +2.63 for TranslateGemma.The cleaning gains for NLLB are roughly half those for either decoder-only LLM, and TranslateGemma has the largest gain despite the highest post-fine-tuning quality.Source: §4.3 · pp. 6–7
• For Gemma-4 Conversation, unfiltered fine-tuning falls below the 3-shot base on BLEU (31.61 vs. 32.54), while cleaned fine-tuning reaches 34.74.Cleaning wins BLEU in every model-domain case except TranslateGemma on Other miscellaneous (−0.09).Source: §4.3 · p. 7
• The ablation supports data quality as a first-order lever for English–Belarusian translation; even datasets considered cleaned upstream retain substantial noise.The authors leave deeper investigation of that residual noise to future work.Source: §4.3 · p. 7

Multilingual translation

• Encoder-decoder MT expanded from per-pair OPUS-MT to M2M-100, NLLB, MADLAD-400, and Omnilingual MT, whose announced target is 1,600 languages.The cited systems progressively broaden multilingual coverage [Fan et al., 2021; NLLB Team et al., 2022; Kudugunta et al., 2023; Alastruey et al., 2026].Source: §5 · p. 7

LLM-based translation

• Off-the-shelf LLMs have lagged dedicated systems on low-resource translation, while fine-tuning recipes such as ALMA and Tower have sought to close that gap [Robinson et al., 2023; Xu et al., 2024; Alves et al., 2024].ALMA combines target-language continued pretraining with bilingual fine-tuning; Tower mixes translation with related tasks.Source: §5 · p. 7

WMT General MT Task

• WMT reports that fine-tuned LLM systems began outperforming standard encoder-decoder approaches in 2024, aligning with this paper’s English–Belarusian finding.LLM systems had become competitive in WMT in 2023.Source: §5 · p. 7

Data quality and filtering

• Web-mined bitext is associated with systematic quality issues, and existing filtering toolkits are largely language-agnostic; this work adds Belarusian-specific orthographic normalization and spellchecking [Kreutzer et al., 2022; Bogoychev et al., 2023].The cited mining approaches include LASER-based corpora and NLLB’s use of distilled LASER 3 encoders [Artetxe and Schwenk, 2019; El-Kishky et al., 2020; Schwenk et al., 2021b,a; NLLB Team et al., 2022].Source: §5 · pp. 7–8

• On BOUQuET, TranslateGemma-12B-IT led every reported metric (32.63 BLEU, 55.36 chrF++, 90.03 COMET), while Gemma-4-12B-IT reached parity with fully fine-tuned NLLB-200-3.3B.Source: §6 · p. 8
• Filtering raised BLEU by +3.62 for Gemma-4 and +4.38 for TranslateGemma, roughly twice NLLB’s +1.99; chrF++ and COMET followed the same architectural split.Source: §6 · p. 8
• The results support data quality as a first-order lever on English–Belarusian translation quality.Source: §6 · p. 8
• Future work includes scaling back-translation beyond 2,300 curated HPLT segments, diversifying training styles, and adding human evaluation and significance testing.Source: §6 · p. 8

• The study did not conduct thorough human evaluation; BLEU may underestimate Belarusian translation quality, and COMET lacks Belarusian meta-evaluation.Source: Limitations · p. 8
• Results lack statistical significance testing, especially relevant to smaller ablation gaps and close model rankings.Source: Limitations · p. 8
• Some larger or potentially more capable models were unevaluated, and no commercial Belarusian translation baseline was tested.Source: Limitations · p. 8
• The fixed few-shot examples may have affected base-LLM performance, but prompt sensitivity was not tested.Source: Limitations · p. 8
• Compute constraints limited LLM fine-tuning to LoRA and the initial base-model screen to 4B parameters; the hyperparameter search was limited.Source: Limitations · p. 8

• Claude and GPT assisted with wording refinement and proofreading.Source: Use of AI assistants · p. 9
• The authors reviewed and edited AI-assisted output and take full responsibility for the submitted work.Source: Use of AI assistants · p. 9
• The authors report that no sensitive or personal data was entered into the tools and that data handling complied with University of Oslo regulations.Source: Use of AI assistants · p. 9

• The benchmark sample comprised 1,013,384 characters from Belarusian Wikipedia articles selected in descending page-length order.Source: A Belarusian tokenisation across… · p. 13
• The sample retained paragraphs classified as Belarusian with alphabetic characters exclusively from the Belarusian alphabet, then passed unchanged to each tokeniser.Source: A Belarusian tokenisation across… · p. 13
• The comparison included tokenisers from Tokeniser Playground and a CLARIN tokeniser made specifically for Belarusian.Source: A Belarusian tokenisation across… · p. 13
• Table 6 reports token counts for the same 1,013,384-character, 263,707-word sample.Source: Table 6 · p. 13