PaperlayerView exact revision on arXiv

Source-linked AI summary

Back-Translation Still Matters: Leveraging Monolingual Data for Low-Resource Machine Translation with LLMs

Divyanshu Bathla, Parag Singla, Jitin Singla

arXiv:2610.18818v2cs.CLcs.AI

TL;DR

Low-resource translation faces weak model competence and scarce, often domain-limited parallel data, raising the question of how to use available monolingual text. The paper adapts instruction-tuned LLMs with back-translation interleaved with supervised updates, and reports gains over parallel-only fine-tuning across low-resource languages and backbones.

  • Problem

    Low-resource languages are sparsely represented in pretraining, and parallel corpora are typically much smaller and often domain-limited; the paper asks how an LLM with a weak prior can use moderate monolingual text for translation.

    Source: §1 · p. 1

  • Method

    BackTransLLM translates monolingual sentences into the other language and trains on the resulting pairs in reverse, interleaving these updates with supervised updates on the small parallel set.

    Source: §1 · p. 2

  • Results

    Back-translation improves translation into the low-resource language by 16–173% XCOMET-XXL over parallel data alone across five low-resource languages and two 3B backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this setting, the pre-LLM back-translation recipe remains useful; the authors recommend beam-decoded pivots and prioritizing monolingual-text quality before domain.

    Source: §5 · p. 9

    The study uses 3B instruction-tuned backbones with about 1K parallel pairs and translates only to and from English; several evaluations cover two languages on one backbone.

    Source: §5 · p. 10

Abstract

from arXiv · show

Pre-trained Large Language Model (LLMs) are effective on many tasks in zero-shot and few-shot settings. For machine translation of low-resource languages, the obvious options are few-shot prompting and fine-tuning on a small parallel set. We ask: what if moderate amounts of low-resource monolingual data are also available? We argue that, rather than continued pretraining, a more effective use of this data is the pre-LLM era technique of back-translation (BT). The current fine-tuned LLM translates monolingual sentences in each language, and the resulting pairs train it in the reverse direction. Updates on the parallel set are interleaved for stable learning. With 1K parallel pairs on Odia and Swahili, BT beats continued pretraining into the low-resource language by 7.7 to 16.4 XCOMET-XXL points in domain. With beam-decoded intermediate translations, it also beats a model with 10 time more parallel data. In-context examples retrieved from the parallel data further improve BT. We also analyze the source, quality and amount of monolingual text. Across five low-resource languages, with Hindi as a high-resource control, and two 3B backbones, BT improves translation into the low-resource language by 16 to 173% XCOMET-XXL over parallel-only fine-tuning. Blind human evaluation on Odia and Sanskrit agrees with the automatic metrics.

1 INTRODUCTION

The paper asks how an instruction-tuned LLM with weak low-resource-language competence can use moderate monolingual data, and tests back-translation against other ways of using it. Across low-resource languages and backbones, back-translation improves translation into the low-resource language over parallel-only fine-tuning.

1 INTRODUCTION

  • Low-resource languages are sparsely represented in pretraining, and their gold-standard parallel corpora are typically two to three orders of magnitude smaller than high-resource corpora and often domain-limited.Open-source instruction-tuned models can therefore have weak or near-zero competence in many low-resource languages.Source: §1 · p. 1
  • The paper asks whether an LLM with a weak prior can use moderate monolingual text for better low-resource translation, and how that text should enter training.Few-shot prompting and fine-tuning on a small parallel set both depend on what the model already knows about the language.Source: §1 · p. 1
  • BackTransLLM interleaves back-translation updates on monolingual text with supervised updates on the small parallel set to keep learning stable.It translates monolingual sentences into the other language, then learns to reconstruct the original sentence from the pivot; a stop-gradient keeps the training target anchored to original text.Source: §1 · p. 2
  • Across five low-resource languages and two 3B backbones, back-translation improves translation into the low-resource language by 16–173% XCOMET-XXL over parallel data alone.Blind human evaluation on Odia and Sanskrit agrees with the automatic metrics.Source: §1 · p. 2
  • On Odia and Swahili, back-translation trails continued pretraining by 7.7 to 16.4 XCOMET-XXL points in domain, while beam-decoded pivots beat a model trained on 10× more parallel data on all 16 measures.Retrieved in-context examples from the parallel data further improve back-translation.Source: §1 · p. 2
  • The experiments compare back-translation with continued pretraining and self-distillation, and examine intermediate-translation decoding, retrieved examples, and monolingual-text source, quality, and amount.The recipes use the same backbone and monolingual text; continued pretraining and BackTransLLM also share the same 1K parallel pairs.Source: §1 · pp. 2–3

2 WAYS TO LEVERAGE MONOLINGUAL DATA

The section contrasts ways to turn monolingual sentences into training examples for adapting an instruction-tuned LLM. Back-translation keeps the human-written sentence as the target, unlike self-distillation, while supervised updates use the small parallel set.

Setting

  • The setting assumes a small parallel set and much larger English and low-resource-language monolingual pools, with an instruction-tuned model fluent in English but limited in the low-resource language.Each recipe fine-tunes on the parallel set with supervised loss.Source: §2 · p. 3

Continued pretraining

  • Continued pretraining treats monolingual text as a language-modeling target before supervised fine-tuning, keeping the loss on human-written text.The cited ALMA recipe uses full-weight continued pretraining followed by fine-tuning on human-written parallel examples.Source: §2 · p. 3

Self-distillation

  • Self-distillation uses monolingual text as input and the model’s own translation as target, repeating the process over cycles.The paper uses it as a control because errors can compound and prior work reported no gains when the base model could not translate the language.Source: §2 · p. 3

Back-translation

  • Back-translation uses a model-generated translation as input and the original monolingual sentence as target, so the model’s errors enter training as input rather than as training targets.The paper’s training cycle alternates supervised steps with four back-translation steps per direction before switching direction.Source: §2 · p. 3, Figure 1 · p. 4

Other related work

  • Other related approaches differ from this setting: some use a stronger external teacher or an already translation-trained NLLB model, while round-trip reconstruction supplies no signal on real target-language text.The paper’s setting adapts a general-purpose LLM with a weak prior using 1K parallel pairs and monolingual text.Source: §2 · pp. 3–4

Back-translation

  • BackTransLLM generates pivots on the fly with its current parameters, alternates back-translation and supervised steps, and uses one model for both directions.A stop-gradient on the pivot makes the back-translation gradient come from predicting original monolingual text.Source: Figure 1 · p. 4

3 BACKTRANSLLM: BACK-TRANSLATION FOR INSTRUCTION-TUNED LLMS WITH WEAK PRIORS

BackTransLLM uses back-translation to give a weak-prior instruction-tuned LLM both a cross-lingual mapping signal and original monolingual text as its prediction target. It interleaves supervised and back-translation updates, with a stop-gradient on generated pivots.

3 BACKTRANSLLM: BACK-TRANSLATION FOR INSTRUCTION-TUNED LLMS WITH WEAK PRIORS

  • Back-translation combines a translation signal with original monolingual text as the target, unlike continued pretraining or self-distillation from poor translations under a weak prior.BackTransLLM retains this anchoring in an instruction-tuned LLM.Source: §3 · p. 4
  • Each training cycle takes one supervised step, then r back-translation steps, alternates direction, and generates pivots on the fly; r=4 is the chosen robust setting.The parallel data is cycled as needed while epochs are defined over the monolingual pools.Source: §3 · p. 4
  • The supervised step predicts the paired target, while each back-translation step predicts the original monolingual sentence from its pivot, with a stop-gradient on that pivot.This makes each back-translation gradient come from predicting original monolingual text.Source: §3 · p. 4
  • Pivot quality bounds what reconstruction can teach; sampling adds noise where the weak-prior model generates poorly, while the study compares it with five-beam search.The compared sampling setting is sharpened nucleus sampling with τ=0.3 and top-p=0.9; test-time decoding is greedy for every system.Source: §3 · p. 4

4 EMPIRICAL RESULTS AND DISCUSSION

The experiments compare ways to use monolingual text for low-resource translation and examine in-context examples, text source and quantity, cross-language and backbone generality, and human judgments. Across five low-resource languages, Hindi as a control, and two 3B backbones, BackTransLLM improves translation into low-resource languages over parallel-only fine-tuning.

4.1 BACK-TRANSLATION USES MONOLINGUAL DATA BEST

  • CPT→SFT gains 16.62 X-XXL points over 1K-SFT into Odia in-domain, while sampled-pivot BackTransLLM leads CPT→SFT into Odia and Swahili.The sampled-pivot margins are 7.67 and 16.39 X-XXL points in-domain, respectively.Source: §4.1 · p. 5
  • With beam pivots, BackTransLLM beats 10K-SFT on all 16 measures and CPT→SFT on 15; self-distillation deteriorates and can collapse into the wrong language.After three cycles into Odia, 99% of self-distilled outputs contain no Odia script.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 IN-CONTEXT LEARNING HELPS EVERY RECIPE

  • ICL improves BackTransLLM by 3.29 and 5.78 X-XXL points into Odia and English, and beam-decoded pivots remain the larger lever.With ICL, BackTransLLM leads CPT→SFT on all four measures; beam-pivot BackTransLLM without ICL beats CPT→SFT with ICL on all four.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 MONOLINGUAL TEXT QUALITY MATTERS AS MUCH AS DOMAIN

  • For Odia, alternate monolingual sources raise X-XXL into the low-resource language from 42.05 to 51.82–66.34, while web text is less reliable into English.IndicCorp-2.0 and CC-100 reach 62.21 X-XXL into Odia.Source: §4.3 · p. 7, Table 3 · p. 8
  • With parallel data fixed, VaqasAI is the better BT source in all eight tested settings, suggesting domain match alone does not explain performance.The margins remain small on the OdiEnCorp-2.0 test set, at 0.6 to 2.5 X-XXL points.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 MORE MONOLINGUAL TEXT HELPS, WITH DIMINISHING RETURNS

  • With 1K parallel pairs, Odia X-XXL rises from 42.05 without monolingual data to 67.9 with 80K sentences, but the 40K-to-80K gain is 1.6 points.The authors use 40K sentences as the default; the pattern also holds at other parallel-data budgets.Source: §4.4 · p. 8

4.5 GAINS HOLD ACROSS LANGUAGES AND BACKBONES

  • BackTransLLM is best in 45 of 48 cells and improves X-XXL into low-resource languages by 16% to 173% over 1K-SFT across five languages and two backbones.With one tenth of the parallel data, it matches or exceeds 10K-SFT into low-resource languages in eight of ten comparisons.Source: §4.5 · p. 8, Table 4 · p. 9

4.6 HUMAN JUDGES AGREE WITH THE METRICS

  • All three blind annotators rank BackTransLLM best for Odia and Sanskrit, followed by 10K-SFT and then 1K-SFT.The evaluation covers 110 held-out source sentences per language.Source: Table 5 · p. 9, §4.6 · p. 9

5 CONCLUSIONS AND LIMITATIONS

The paper concludes that back-translation is effective for low-resource translation with bounded monolingual data, while noting that its experiments cover a narrow model, data, and language-pair regime.

5 CONCLUSIONS AND LIMITATIONS

  • Across five low-resource languages and two 3B backbones, back-translation improves translation into the low-resource language, and human judges agree on Odia and Sanskrit.With beam-decoded pivots, it also beats continued pretraining and ten times more parallel data; retrieved in-context examples further improve it.Source: §5 · p. 9
  • The authors recommend beam-decoded pivots and prioritizing monolingual-text quality over domain match in this regime.Source: §5 · p. 9
  • The experiments use 3B instruction-tuned backbones, about 1K parallel pairs, and translation only to and from English.Recipe comparisons, the in-context learning study, and human evaluation each cover two languages on one backbone.Source: §5 · p. 10
  • The ALMA-style and CycleDistill-style baselines are reimplementations at this scale, and self-distillation may work with stronger priors or larger models.Source: §5 · p. 10

AI USE STATEMENT

Generative AI tools assisted with code and manuscript language editing, while the authors report that they did not use them to develop or produce the research.

AI USE STATEMENT

  • Generative AI tools assisted with writing and debugging experiment scripts, data processing, and figure generation.Source: AI USE STATEMENT · p. 10
  • The tools also assisted with manuscript language editing, including grammar, wording, and sentence flow.Source: AI USE STATEMENT · p. 10
  • The authors report that AI tools did not generate research ideas, design methods or experiments, produce or alter results, or write the related-work analysis.Authors reviewed and verified the AI-assisted code and reviewed edited text for accuracy and consistency.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION AND REPRODUCIBILITY DETAILS

The appendix documents data, prompting, metrics, and training configurations, then reports detailed results for recipe, interleaving, in-context learning, and monolingual-source comparisons.

A IMPLEMENTATION AND REPRODUCIBILITY DETAILS

  • The implementation uses 40K monolingual sentences per language, target-token-only training loss, and five automatic metrics, including XCOMET and ChrF++.Under the LLaMA tokenizer, the Odia–English pools contain 9.69M tokens and the Swahili–English pools 3.51M; systems are greedily decoded at test time.Source: A IMPLEMENTATION AND REPRODUCIBILITY… · pp. 15–16

B TRAINING RECIPES

  • The training recipes share data, prompts, metrics, and settings across backbones, while differing in what is trained, on which data, and for how long.ALMA-style training applies full-weight continued pretraining before LoRA fine-tuning on 1K pairs; self-distillation uses three cycles with a 3B model as its own teacher.Source: B TRAINING RECIPES · p. 17

C RECIPE COMPARISON: FULL RESULTS

D INTERLEAVING RATIO r: SFT VS. BT STEPS

  • Odia peaks at r=2 and Sanskrit at r=4; the study uses r=4, which stays within 2.7 points of Odia’s best ratio on every column.A supervised step is followed by r back-translation steps in the same direction.Source: D INTERLEAVING RATIO r: SFT VS. BT STEPS · p. 18

E IN-CONTEXT LEARNING

  • BackTransLLM with retrieved examples improves every metric in both directions for Odia and Amharic in-domain, but gains are smaller or inconsistent on Flores-200.The examples come from the 1K seed; prompts drop examples when they would exceed 2,048 tokens.Source: E IN-CONTEXT LEARNING · p. 19

F MONOLINGUAL SOURCE: FULL RESULTS

  • The in-corpus monolingual text is best on every metric in both directions for Odia and Assamese, while other sources also improve all five metrics over 1K-SFT into the low-resource language.Into English, alternative sources give mixed results: CC-100 falls below 1K-SFT on all five Odia metrics.Source: F MONOLINGUAL SOURCE: FULL RESULTS · p. 19

G MONOLINGUAL TEXT QUALITY VERSUS DOMAIN: FULL RESULTS

The section compares monolingual corpora as back-translation sources and as continued-pretraining data, finding that the preferred source depends on the training approach and evaluation setting.

G MONOLINGUAL TEXT QUALITY VERSUS DOMAIN: FULL RESULTS

  • With parallel data fixed, VaqasAI is ahead of OdiEnCorp-2.0 on X-XXL and X-XL in all eight held-out back-translation settings, including the OdiEnCorp-2.0 test set.On that test set, the X-XXL margins are 0.6 to 2.5 points and have not been tested for significance.Source: G MONOLINGUAL TEXT QUALITY VERSUS… · pp. 19, 22
  • On Flores-200, VaqasAI back-translation leads on every metric across both parallel corpora and both translation directions.Source: G MONOLINGUAL TEXT QUALITY VERSUS… · p. 22
  • For back-translation, the findings suggest corpus quality matters at least as much as domain, whereas continued pretraining on OdiEnCorp-2.0 beats VaqasAI on two evaluation sets.The continued-pretraining comparison uses VaqasAI parallel data and favors OdiEnCorp-2.0 on every metric on the OdiEnCorp-2.0 test set and Flores-200.Source: G MONOLINGUAL TEXT QUALITY VERSUS… · p. 22

H SCALING PARALLEL AND MONOLINGUAL DATA

The scaling analysis finds that moderate monolingual data improves translation, with the largest gains into Odia and diminishing returns beyond 40K sentences. More monolingual data does not uniformly outperform using more parallel data, particularly into English.

H SCALING PARALLEL AND MONOLINGUAL DATA

  • With 100 parallel pairs, 40K monolingual sentences lift EN→XX X-XXL to 57.66, versus 42.05 for 1K-SFT and 47.71 for CPT→SFT with 100 pairs.The advantage is specific to translation into Odia; CPT→SFT leads into English, 52.13 against 46.50.Source: H SCALING PARALLEL AND MONOLINGUAL DATA · p. 22
  • X-XXL rises 5.3 points from 20K to 40K monolingual sentences, but only 1.6 points from 40K to 80K for Odia EN→XX, motivating the 40K default.On Flores-200, X-XXL rises from 20K to 40K and falls slightly from 40K to 80K.Source: H SCALING PARALLEL AND MONOLINGUAL DATA · p. 22, Figure 2 · p. 25
  • At 40K monolingual sentences, EN→XX X-XXL increases with parallel data from 57.66 with 100 pairs to 66.34 with 1K and 72.04 with 10K pairs.The scaling experiments also compare 100, 1K, and 10K parallel pairs and multiple monolingual-data amounts against SFT and CPT→SFT baselines.Source: H SCALING PARALLEL AND MONOLINGUAL DATA · p. 22, Table 19 · p. 25
  • Adding 40K monolingual sentences to 10K parallel pairs yields lower X-XXL than 10K-SFT: 57.48 versus 61.51 in domain and 35.93 versus 47.07 on Flores-200.Source: H SCALING PARALLEL AND MONOLINGUAL DATA · p. 25

I LANGUAGES AND BACKBONES: FULL HELD-OUT RESULTS

On held-out tests, BackTransLLM improves translation into the low-resource language across all tested language and backbone combinations, often exceeding 10K-SFT on selected metrics. On Flores-200, gains over 1K-SFT are less consistent, especially into English with sampled pivots.

I LANGUAGES AND BACKBONES: FULL HELD-OUT RESULTS

  • BackTransLLM improves over 1K-SFT on all five metrics into the low-resource language across both backbones and all six languages; it exceeds 10K-SFT on X-XXL and ChrF++ in eight of ten runs.Into English, it improves over 1K-SFT in nine of twelve combinations, while 10K-SFT has higher X-XXL in all ten comparisons.Source: I LANGUAGES AND BACKBONES: FULL… · p. 25, Table 21 · p. 26

J LANGUAGES AND BACKBONES: FULL FLORES-200 RESULTS

  • On Flores-200, BackTransLLM improves over 1K-SFT on all five metrics into the low-resource language in ten of twelve combinations, but exceeds 10K-SFT on X-XXL in only five of ten.With sampled pivots, it remains below 1K-SFT on English-direction X-XXL in nine of twelve combinations; beam-decoded pivots exceed both 1K-SFT and 10K-SFT on Odia and Swahili.Source: J LANGUAGES AND BACKBONES: FULL… · p. 27

K SEED ROBUSTNESS AND SIGNIFICANCE (ODIA)

Across three Odia runs, BackTransLLM’s advantage over 10K-SFT depends on the metric and test used. Qualitative examples show both systems recovering source meaning where the 1K-SFT output is often fluent but inadequate or partly code-mixed.

K SEED ROBUSTNESS AND SIGNIFICANCE (ODIA)

  • Across three seeds, standard deviations are at most 1.85 points, exceeding the 0.10-point X-XXL gap between BackTransLLM (66.52) and 10K-SFT (66.42) into Odia.Source: K SEED ROBUSTNESS AND SIGNIFICANCE… · p. 27
  • Into Odia, BackTransLLM significantly leads 10K-SFT on X-XL, BERTScore, and BLEU, but not on X-XXL (p=0.67) or ChrF++ (Holm-adjusted p=0.12).A paired bootstrap favors BackTransLLM on mean ChrF++ by 0.72 points (95% CI 0.26 to 1.18), so that finding depends on the test; into English, 10K-SFT is significantly ahead on every metric.Source: K SEED ROBUSTNESS AND SIGNIFICANCE… · p. 27

L QUALITATIVE EXAMPLES

  • In selected Odia and Swahili examples where both systems improve over 1K-SFT on ChrF++ and X-XXL, BackTransLLM and 10K-SFT recover source meaning despite often fluent but inadequate or code-mixed 1K-SFT outputs.Source: L QUALITATIVE EXAMPLES · p. 27

M HUMAN EVALUATION PROTOCOL

The human evaluation used anonymous, reference-free rankings of three Odia and Sanskrit translation systems, with annotators agreeing above chance. Conditional translation perplexity broadly follows translation quality, but exceptions show it need not track reference-based quality.

M HUMAN EVALUATION PROTOCOL

  • Three scholars per language ranked 110 English-source sentences’ outputs from 1K-SFT, 10K-SFT, and BackTransLLM in random order, without system names or reference translations.Odia annotators were native speakers; Sanskrit annotators were language experts.Source: M HUMAN EVALUATION PROTOCOL · pp. 27, 30
  • Figures 3 and 4 show Odia and Swahili examples where BackTransLLM and 10K-SFT both improve over 1K-SFT on ChrF++ and X-XXL.The Odia example includes a 1K-SFT output with high X-XXL despite low ChrF++, motivating a model-independent complement to the learned metric.Source: Figure 3 · p. 29, Figure 4 · p. 29
  • Annotators’ mean per-sentence Kendall’s W is 0.66 for Odia and 0.58 for Sanskrit, above 0.33 for random rankings; Fleiss’ κ on the best-ranked system is 0.37 and 0.25.Two annotators rank a pair of systems alike in 72.5% of Odia and 66.7% of Sanskrit cases, versus 50% by chance.Source: M HUMAN EVALUATION PROTOCOL · p. 30

N CONDITIONAL TRANSLATION PERPLEXITY

  • Conditional translation perplexity is the reference translation’s perplexity given the source prompt, measuring translation modelling rather than target-language fluency alone.Because target-token counts vary by direction, absolute perplexities should be compared only within a direction.Source: N CONDITIONAL TRANSLATION PERPLEXITY · pp. 30–31, Table 24 · p. 31
  • BackTransLLM has the lowest conditional translation perplexity into English in all twelve language-backbone combinations and the lowest or second-lowest into the low-resource language in eleven of twelve.Despite better translation quality, its EN→XX perplexity exceeds 1K-SFT for Amharic and mildly for Swahili with LLaMA, so conditional perplexity need not track reference-based quality.Source: N CONDITIONAL TRANSLATION PERPLEXITY · p. 31, Table 25 · p. 31

Related papers

Loading 2610.18818v2…