Source-linked AI summary
Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation
Hanxu Hu, Zdeněk Šnajdr, Pinzhen Chen, Jannis Vamvas, Rico Sennrich
TL;DR
Extremely low-resource translation requires models to use linguistic context beyond memorized languages, but existing post-training and in-context methods have limited zero-shot transfer. The paper trains LLMs with RL using chrF rewards on context-rich translation prompts, finding better generalization to unseen languages than in-context learning or SFT. The results support outcome-based RL as a way to teach language learning from context.
Problem
Existing post-training and in-context approaches can overfit specific languages, motivating a language-independent way to use linguistic knowledge for unseen-language translation.
Method
The method uses reinforcement learning to train LLMs on translations conditioned on linguistic context and dictionary entries, scoring final translations with chrF.
Results
RL-trained models generalize substantially better to unseen languages than SFT counterparts and exploit linguistic context rather than memorizing training languages.
Takeaways & Limitations
Outcome-based RL can extend beyond math and coding reasoning tasks as a recipe for learning language-independent contextual leveraging.
Takeaways & Limitations
The study reports chrF++ without human evaluation, and absolute unseen-language performance still trails higher-resource languages.
Abstract
from arXiv · showhide
Prior work has shown that large language models (LLMs) can translate unseen or low-resource languages by undergoing continued training or even by encoding a grammar book in their context. However, both methods typically overfit specific languages, with limited zero-shot transfer at test time. To translate extremely low-resource languages at scale, we argue that LLMs must acquire the meta-skill of utilizing in-context linguistic knowledge rather than memorizing specific languages. In this paper, we propose a reinforcement learning (RL) approach to unseen language translation given rich linguistic context, using a surface-level translation metric (chrF) as the reward. Empirically, despite the lightweight reward, our RL-trained models effectively extract and apply relevant linguistic information from the provided context, leading to better translations on completely unseen languages than in-context learning or supervised fine-tuning. Our analyses suggest that outcome-based RL can extend beyond conventional reasoning tasks like math and coding to serve as a recipe for language learning from context.
1 Introduction
Extremely low-resource translation exposes the difficulty of applying linguistic reasoning outside the training distribution. The paper proposes RL to teach LLMs a language-independent meta-skill for using rich linguistic context rather than memorizing languages.
- Extremely low-resource and unseen-language translation tests LLM reasoning over linguistic patterns when pre-training data is scarce.The setting also has implications for documenting and preserving endangered languages lacking large parallel corpora.
- Existing approaches either post-train on language-specific data or use in-context examples, but both can overfit particular languages.
- The proposed target is a transferable meta-skill that generalizes across languages by reasoning over grammars, morphological paradigms, lexicons, and other contextual resources.
- RL trains LLMs to translate new languages from linguistic context and dictionary entries, using translation quality such as chrF as an outcome reward.The task treats translation given grammar as a verifiable, context-dependent reasoning problem.
- RL can elicit contextual leveraging and improve generalization to unseen languages, while controlled comparisons show better unseen-language performance than SFT.
2 Related Work
Prior work uses documentation, dictionaries, rules, or supervised adaptation to support low-resource translation, but generalization remains limited. This paper frames contextual resource use as meta-learning and applies RLVR to heterogeneous linguistic evidence.
- Earlier methods provide grammar books, dictionaries, analyzed text, rules, or finite-state transducers to guide translation without necessarily updating model parameters.
- Fine-tuning and chain-of-thought can improve translation from grammar descriptions, but generalizability is limited for complex linguistic phenomena.
- Recent RL-based machine-translation work uses rewards such as BLEU, COMET-Kiwi, COMET, and format signals under GRPO.
- The paper frames translation as meta-learning over a contextual support set of linguistic resources used at inference time.
- Unlike supervised meta-learning, the approach uses RLVR to teach models to exploit qualitatively heterogeneous in-context resources for low-resource translation.
3 Methodology
The method builds linguistically informed prompts and trains language models with outcome-based RL on translation quality. Its dataset combines grammar-derived and parallel-corpus examples across seen, related, and unseen evaluation settings.
- Dataset construction: The corpus construction covers language selection, parallel-sentence extraction, Romansh treatment, synthetic dictionaries, and assembled prompt structure.
- Dataset construction: The study investigates 14 very low-resource languages selected by data-sourcing pipeline, including eight languages with grammar-book-extracted parallel data.
- Dataset construction: Romansh varieties enable evaluation across language relatedness, while other languages are selected from openly licensed grammar books available in PDF and LATEX formats.
- Dataset construction: Translated examples are extracted from grammars, with Romansh instead using sampled back-translated parallel data; pairs are length-filtered and duplicated bidirectionally.
- Prompt construction: Synthetic dictionaries are generated for languages lacking sufficient resources, and variants are selected per language and direction by resulting chrF.
- Prompt construction: Prompts combine linguistic descriptions, retrieved dictionary entries, parallel sentences, grammar passages, and a source sentence, with LCS retrieval and three- or five-example conditions.
- Dataset construction: The dataset spans 18 languages from 10 families, with 32,335 training pairs and 2,699 test pairs partitioned into seen, similar, and unseen evaluation directions.
- RL formulation: Each RL instance is a source sentence, reference translation, and meta-linguistic context; the policy generates reasoning followed by a translation, but only the translation is scored.
4 Experiments
The experiments compare SFT and RL across seen, related held-out, and entirely unseen languages under full and absent retrieval context. SFT is stronger on familiar languages, while RL generalizes better to unrelated languages when contextual resources are available.
- Evaluation setup: The evaluation covers seen Romansh varieties, held-out Romansh varieties, and five unrelated unseen languages, using full retrieval or task-instruction-only context.The unseen set comprises Kalamang, Dinka, Wolof, Guarani, and Kachin; results use chrF on a 0–1 scale.
- Comparison of SFT and RL: SFT outperforms RL on seen and held-out Romansh, including Qwen SFT’s 0.60 versus RL’s 0.52 on seen varieties and 0.55 versus 0.48 on held-out varieties.LLaMA shows the same seen-language pattern, with SFT scoring 0.56 versus RL’s 0.48.
- Comparison of SFT and RL: Qwen RL averages 0.27 on five unseen languages, versus 0.09 for SFT and 0.18 for the untuned base model.Llama-3.2 shows the same ordering: RL 0.24, SFT 0.09, and base 0.14; RL exceeds SFT on each unseen language individually.
- Context dependence: Removing retrieval context exposes different strengths: Qwen SFT falls from 0.60 to 0.46 on seen languages, while RL falls from 0.52 to 0.30.On unseen languages, RL’s advantage appears only with context, reaching 0.27 versus 0.12 under the reported comparison.
- Context ablation: 8.4 chrF is lost when the bilingual dictionary is removed on both seen Romansh and En→Kalamang, from 0.5324 to 0.4483 and 0.3464 to 0.2626.The dictionary has the largest impact because it provides direct word-level grounding.
- Context ablation: −7.3 chrF is lost on OOD En→Kalamang when parallel sentences are removed, compared with −1.0 chrF on seen Romansh; grammar removal causes the smallest drops.Grammar removal produces −0.8 chrF on Puter/Vallader and −1.5 on En→Kalamang.
- Context dependence: Test-time context dominates training-time context: no/full beats full/no in every panel, including 0.28 versus 0.17 on En→Kalamang.Training with context further improves En→Kalamang, with full/full reaching 0.35 versus 0.28 for no/full.
- Training rewards: Full, No-dict, and Task-only runs rank identically in reward trajectories, reaching 0.68, 0.62, and approximately 0.29 in chrF training reward by step 50.Task-only plateaus early, No-dict continues improving past step 100, and Full sustains reward growth throughout training.
5 Conclusion
The paper proposes reinforcement learning for low-resource translation that exploits in-context linguistic information. RL-trained LLMs generalize better to unseen languages than SFT models and learn to leverage context rather than memorize training languages.
- The paper proposes reinforcement learning that trains LLMs to exploit in-context linguistic information for low-resource translation.
- RL-trained LLMs generalize substantially better to unseen languages than their SFT counterparts.
- The results indicate that the models learn to leverage linguistic context rather than memorize the specific languages seen during training.
Limitations
The evaluation relies on chrF++ without human evaluation, while unseen-language performance remains below that of higher-resourced languages.
- The paper reports chrF++ but does not conduct human evaluation of fluency and adequacy.The authors leave human evaluation to future work and treat automatic scores as reliable signals of relative improvement across methods.
- Absolute performance on unseen languages still lags behind performance on higher-resourced languages.The authors identify headroom for richer in-context evidence and stronger context-utilization signals.
A Hyperparameters
The appendix specifies full-parameter supervised fine-tuning and GRPO reinforcement-learning configurations, including optimization, batching, rollout, and distributed-training settings.
- Full-parameter fine-tuning uses DeepSpeed ZeRO-2, a 1e−5 learning rate, cosine scheduling, 10% warmup, and effective batch size 128.The batch size is formed from 4 samples per device and 8 gradient-accumulation steps.
- GRPO uses chrF/100 as the outcome reward, a 1e−6 learning rate, batch size 64, and n = 8 rollouts per prompt at temperature 1.0.The configuration also specifies clip ratio 0.25, KL coefficient 1e−4, and entropy coefficient 1e−3.
- RL training uses FSDP with SGLang and TP=4 for rollout and ablation variants.
B LLM Usage Declaration
The authors used AI for manuscript editing and for generating synthetic bilingual dictionary entries, while declaring the research itself original and reviewing edited text.
- AI was used during manuscript writing to edit language and polish the overall text.
- GPT-5 mini generated synthetic bilingual dictionary entries for languages lacking sufficient dictionary resources.Because no authors spoke the target languages, the entries were not manually validated; prompt variants were selected using downstream chrF.
- The authors declare that ideation, methodology, experiment execution, and analysis are original work.They state that all AI-assisted text editing was carefully reviewed for accuracy.
C Grammar Books Used For Data Curation
The paper curates grammar-book resources from published and print sources to extract translation examples and provide linguistic context for translation prompts.
- Eight grammars in the upper block come from Language Science Press, with CC BY 4.0 licensing and LaTeX sources used to extract parallel examples.
- Two Romansh idiom grammars in the lower block come from Lia Rumantscha print volumes, so examples were extracted from printed text rather than LaTeX.
- The Romansh→German prompt combines linguistic introduction, instructions, dictionary entries, parallel sentences, a grammar excerpt, and a closing instruction.
- The Kalamang→English prompt includes dictionary entries, parallel sentence examples, and grammar passages, with the same structure applied to other endangered languages when dictionaries exist.