Source-linked AI summary
No One-Size-Fits-All: Building Systems For Translation to Bashkir, Kazakh, Kyrgyz, Tatar and Chuvash Using Synthetic And Original Data
Dmitry Karpov
TL;DR
The paper addresses machine translation for five low-resource Turkic language pairs by comparing fine-tuning and prompting approaches. Results indicate that the strongest approach varied by language-resource setting, with LoRA fine-tuning reaching chrF++ 49.71 for Kazakh and retrieved-example prompting reaching 39.47 for Chuvash.
Problem
Machine translation for five low-resource Turkic language pairs remains challenging under data-scarce conditions.
Method
The study compares fine-tuning nllb-200-distilled-600M with LoRA on synthetic data against zero-shot and retrieval-augmented prompting of large language models.
Results
Performance varied by language: LoRA fine-tuning achieved chrF++ 49.71 for Kazakh and 46.94 for Bashkir, while retrieved-example prompting achieved 39.47 for Chuvash.
Takeaways & Limitations
Fine-tuning was most promising for relatively well-resourced languages, whereas similar-phrase prompting was most effective for extremely resource-scarce Chuvash.
Takeaways & Limitations
Further exploration of one promising approach was not pursued because of limited computational resources.
Abstract
from arXiv · showhide
We explore machine translation for five Turkic language pairs: Russian-Bashkir, Russian-Kazakh, Russian-Kyrgyz, English-Tatar, English-Chuvash. Fine-tuning nllb-200-distilled-600M with LoRA on synthetic data achieved chrF++ 49.71 for Kazakh and 46.94 for Bashkir. Prompting DeepSeek-V3.2 with retrieved similar examples achieved chrF++ 39.47 for Chuvash. For Tatar, zero-shot or retrieval-based approaches achieved chrF++ 41.6, while for Kyrgyz the zero-shot approach reached 45.6. We release the dataset and the obtained weights.
1 Introduction
Machine translation for low-resource languages remains challenging, motivating investigation of multiple approaches for five Turkic language pairs in data-scarce conditions.
- The study investigates machine translation for Russian-Bashkir, Russian-Kazakh, Russian-Kyrgyz, English-Tatar, and English-Chuvash.
2 Making The Data
Because high-quality parallel data was limited, the study combined multiple existing corpora with synthetic translations, expanded each training set substantially, and released the resulting multilingual dataset.
- Limited high-quality parallel data led the study to combine datasets from multiple existing sources.
- Synthetic translations were generated through sequential English-to-Russian and Russian-to-Turkic translation, then filtered against test-set phrases.
- 2,457,344 samples formed the augmented training set for each language pair.
- MASSIVE translations were reserved for prompting-based solutions rather than NLLB training because they were obtained later.
- The English-Chuvash prompting dataset reached 6.7 million pairs after adding additional Chuvash-English corpora, while corresponding Tatar translations were not used in Tatar submissions.
- The resulting YaTURK-7lang dataset was released with six-language translations and markings for data used only in the Chuvash solution.
3 Kazakh and Bashkir: Where LoRA And Knowledge Transfer Shined
The study compared multilingual and single-task NLLB fine-tuning, finding that multilingual training followed by adapter fine-tuning outperformed single-task fine-tuning and produced strong Kazakh and Bashkir results.
- The study used facebook/nllb-200-distilled-600M as the base model for fine-tuning.
- Two training modes were compared: separate two-epoch language fine-tuning and multilingual one-epoch training followed by language-specific LoRA adapters.
- Multilingual training followed by LoRA adapters outperformed single-task fine-tuning, suggesting knowledge transfer between related Turkic-language tasks.
- The promising approach was not pursued further because of limited computational resources.
- 49.71 chrF++ was achieved for Kazakh and 46.94 chrF++ for Bashkir in the submitted results.
4 Chuvash and Tatar: Exploring Prompting
Prompting experiments used retrieval-based examples and several large language models, with outcomes varying by language: retrieval helped Chuvash, while Tatar and Kyrgyz favored other prompting settings.
- The retrieval-based system indexed source-language phrases and appended similar source-translation examples to each language-model prompt.
- The Chuvash final experiment used TOP_N=7000, while the prompt was truncated to 129,800 tokens for DeepSeek-V3.2.
- The experiments prompted DeepSeek-R1, DeepSeek-N1, MiMoV2, Gemma3, and DeepSeek-V3.2.
- 39.47 chrF++ was achieved on the Chuvash test set by retrieval-augmented prompting with DeepSeek-V3.2.
- 43.66 was the strongest reported English-Tatar result, achieved by zero-shot DeepSeek-V3.2, while larger contexts and filtering heuristics reduced performance.
5 Bashkir, Kazakh and Kyrgyz: Where Prompting Failed
For Kyrgyz, zero-shot prompting achieved the highest score, while larger context windows generally reduced chrF++ and further improvements were not pursued.
- 45.61 chrF++ was achieved on the Kyrgyz test set using submitted DeepSeek-V3.2 and MiMoV2 results.The highest Kyrgyz score came from zero-shot prompting of MiMoV2.
- 46.61 to 45.33 chrF++ dropped when MiMoV2's context window expanded to 130,000 characters and 7,000 examples.
- 39.55 to 33.31 and 47.54 to 42.76 chrF++ dropped for Bashkir and Kazakh, respectively, under the expanded MiMoV2 context.
- 41.59 to 41.61 chrF++ was the insignificant Russian-Bashkir improvement from enlarging DeepSeek-R1's context window to 80,000 characters and 1,000 examples.
6 Stacking The Results
Semantic-similarity selection and Kazakh stacking did not improve validation performance, underscoring the difficulty of identifying the best translation among multiple candidates.
- Minor deterioration occurred when LaBSE semantic similarity selected Kazakh and Kyrgyz translations compared with the best single submission.LaBSE supports both Kazakh and Kyrgyz, but selection still underperformed the best individual submission.
- The most probable translation among several good candidates was not necessarily the best one, making low-resource machine-translation evaluation difficult.
- 49.93 to 49.08 validation chrF++ declined after stacking Kazakh LoRA, zero-shot, and finetuning results.The stacking result remained the second-best Kazakh result, but this branch was not explored further.
7 Discussion
The most effective approach varied by language-resource conditions: synthetic-data finetuning favored Bashkir and Kazakh, similar-example prompting favored Chuvash, and zero-shot models remained strongest for Kyrgyz.
- Synthetic-data finetuning was the most promising explored approach for the relatively well-resourced Bashkir and Kazakh languages.
- Prompting with the most similar phrases proved most effective for Chuvash, where pretraining data was extremely scarce.
- Tatar results were ambiguous, while zero-shot models could not be outperformed for Kyrgyz.
- Retrieved similar-phrase prompting via ANNOY may work for very resource-scarce languages with poor zero-shot performance, whereas finetuning may suit languages with better zero-shot performance.Finetuning models pretrained specifically on low-resource languages remained unexplored future work.
8 Conclusion
The study evaluates five Turkic translation pairs with finetuning and prompting approaches, reports pair-specific chrF++ results, and releases the dataset and obtained weights.
- Five translation pairs were studied: Russian-Bashkir, Russian-Kazakh, Russian-Kyrgyz, English-Tatar, and English-Chuvash.
- 49.71 chrF++ for Kazakh and 46.94 for Bashkir came from LoRA finetuning nllb-200-distilled-600M on synthetic data.
- 39.47 chrF++ for Chuvash came from prompting DeepSeek-V3.2 with retrieved similar examples.
- Tatar reached chrF++ 41.6 with zero-shot or retrieval-based approaches, while Kyrgyz reached 45.6 with the zero-shot approach.
- The dataset and obtained model weights were released.