PaperlayerView exact revision on arXiv

Source-linked AI summary

TranslateGemma Technical Report

Mara Finkelstein, Isaac Caswell, Tobias Domhan, Jan-Thorsten Peter, Juraj Juraska, Parker Riley, Daniel Deutsch, Geza Kovacs, Cole Dilanni, Colin Cherry, Eleftheria Briakou, Elizabeth Nielsen, Jiaming Luo, Kat Black, Ryan Mullins, Sweta Agrawal, Wenda Xu, Erin Kats, Stephane Jaskiewicz, Markus Freitag, David Vilar

arXiv:2601.09012v3cs.CLcs.AI

TL;DR

Strong open machine-translation models can support transparency, reproducibility, and community-driven innovation. TranslateGemma fine-tunes Gemma 3 with supervised learning and reinforcement learning, and evaluations report consistent gains over baseline models, with smaller models often matching or exceeding larger baselines.

  • Problem

    Strong open machine-translation models are needed to support transparency, reproducibility, and community-driven innovation in a field advanced by LLMs.

    Source: §1 · p. 1

  • Method

    TranslateGemma adapts Gemma 3 for machine translation through supervised fine-tuning on diverse parallel data followed by reinforcement learning with an ensemble of reward models.

    Source: §1 · p. 1, §7 · p. 7

  • Results

    On WMT24++ across 55 language pairs, TranslateGemma shows consistent gains over baseline Gemma 3 in MetricX and Comet22, while smaller fine-tuned models often match or exceed larger baselines.

    Source: §7 · p. 7

  • Takeaways & Limitations

    The released open models provide machine-translation tools for research and application development.

    Source: §7 · p. 7

    The authors hypothesize that the 27B model benefited more from the languages seen during SFT, but report no direct experimental confirmation.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

We present TranslateGemma, a suite of open machine translation models based on the Gemma 3 foundation models. To enhance the inherent multilingual capabilities of Gemma 3 for the translation task, we employ a two-stage fine-tuning process. First, supervised fine-tuning is performed using a rich mixture of high-quality large-scale synthetic parallel data generated via state-of-the-art models and human-translated parallel data. This is followed by a reinforcement learning phase, where we optimize translation quality using an ensemble of reward models, including MetricX-QE and AutoMQM, targeting translation quality. We demonstrate the effectiveness of TranslateGemma with human evaluation on the WMT25 test set across 10 language pairs and with automatic evaluation on the WMT24++ benchmark across 55 language pairs. Automatic metrics show consistent and substantial gains over the baseline Gemma 3 models across all sizes. Notably, smaller TranslateGemma models often achieve performance comparable to larger baseline models, offering improved efficiency. We also show that TranslateGemma models retain strong multimodal capabilities, with enhanced performance on the Vistra image translation benchmark. The release of the open TranslateGemma models aims to provide the research community with powerful and adaptable tools for machine translation.

1. Introduction

TranslateGemma is an open Gemma 3 variant refined for machine translation through supervised fine-tuning and reinforcement learning. It shows substantial gains across 55 language pairs and retains multimodal capabilities, with improved image-translation performance.

  • The report motivates open machine-translation models as a way to support transparency, reproducibility, and community-driven innovation.Source: §1 · p. 1
  • The report presents TranslateGemma as an open variant of the Gemma 3 foundation model (Gemma Team, 2025), specifically enhanced for machine translation.Source: §1 · p. 1
  • TranslateGemma improves translation quality over Gemma 3 through supervised fine-tuning on parallel data and reinforcement learning from human and model-based feedback.The fine-tuning data blend human-translated and synthetic texts, while reward models optimize translation quality.Source: §1 · p. 1
  • TranslateGemma demonstrates substantial gains on WMT25 and WMT24++ across 55 language pairs.Source: §1 · p. 1
  • TranslateGemma retains Gemma 3’s multimodal capabilities, and its text-translation enhancements also improve image-translation performance on Vistra.Source: §1 · p. 1

2. Training data

TranslateGemma’s training data combines synthetic and human-translated parallel data, with most data shared between SFT and RL. The synthetic data pipeline selects promising source segments, while additional data supports lower-resource languages and generic instruction following.

2.1. Synthetic Gemini-Generated Translation Data

  • MADLAD-400 monolingual data supports synthetic translation generation, targeting up to 10K examples per language pair to improve translation quality.The approach uses sources bucketed by length and samples one million segments per language pair for selection.Source: §2.1 · pp. 1–2
  • Sources with the greatest MetricX 24-QE improvement from a sampled versus greedy Gemini 2.5 Flash translation are selected as a low-cost proxy for 128-sample QE decoding.For selected sources, 128 translations are generated and MetricX 24-QE selects the best; outputs include individual sentences and text blobs up to 512 tokens, followed by formatting filtering.Source: §2.1 · p. 2

2.2. Human-Generated Translation Data

  • To broaden script coverage in lower-resource languages, training also uses human-generated parallel data from SMOL, covering 123 languages, and GATITOS, covering 170.Source: §2.2 · p. 2

2.3. Language distribution

  • RL uses the same translation data as SFT except that SMOL and GATITOS are used only for SFT; the language proportions are shown in Figure 1.Source: §2.3 · p. 2

2.4. Generic Instruction-Following Data

  • SFT includes 30% generic instruction-following data from the original Gemma 3 mixture to limit translation-task overfitting and retain general instruction-following capabilities.Source: §2.4 · p. 2

3. Supervised Fine-Tuning

TranslateGemma’s supervised fine-tuning starts from Gemma 3 checkpoints and combines human and synthetic parallel data with generic instruction-following data. Training updates all parameters except embeddings, which are frozen to help translation in languages and scripts absent from the SFT mix.

3. Supervised Fine-Tuning

  • SFT begins from released Gemma 3 27B, 12B, and 4B checkpoints and uses human-generated and Gemini-generated parallel data plus generic instruction-following data.Parallel data is described in Section 2.Source: §3 · p. 2
  • Training uses Kauldron SFT tooling, AdaFactor with a 0.0001 learning rate, batch size 64, and 200k steps.All model parameters are updated except the embedding parameters.Source: §3 · p. 2
  • Embedding parameters are frozen because preliminary experiments indicated this helped translation for languages and scripts not covered by the SFT data mix.Source: §3 · p. 2

4. Reinforcement Learning

TranslateGemma applies reinforcement learning to the SFT checkpoint with an ensemble of reward models to improve translation quality, using both sequence-level and token-level signals.

4. Reinforcement Learning

  • RL on the SFT checkpoint used an ensemble of reward models to further boost translation quality.The ensemble combined translation-focused rewards with naturalness and generalist reward models.Source: §4 · pp. 2–3
  • The ensemble included MetricX-24-XXL-QE (Juraska et al., 2024) and Gemma-AutoMQM-QE; MetricX scores were rescaled as 5.0 − score for rewards, making higher scores better, with an empty reference as QE input.Gemma-AutoMQM-QE was trained on MQM ratings data from WMT 2020–2023.Source: §4 · p. 3
  • ChrF was the only reward model to use synthetic references, and its scores were doubled to match the scale of other rewards.ChrF is a lexical-overlap translation metric.Source: §4 · p. 3
  • The ensemble also used an in-house Naturalness Autorater and a generalist reward model adapted from Gemma 3 post-training (Gemma Team, 2025).The Autorater penalized machine-translation spans that sounded non-native, provided unnaturalness did not stem from an unnatural source input.Source: §4 · p. 3
  • Token-level advantages were added to sequence-level rewards, enabling span-level signals for finer credit assignment and training efficiency.AutoMQM and the Naturalness Autorater supplied the span-level signals; combined advantages were batch-normalized.Source: §4 · p. 3

5. Automatic Evaluation

TranslateGemma improves automatic text-translation metrics over Gemma 3 across model sizes and language pairs, with smaller fine-tuned models matching or surpassing larger baselines. It also retains image-translation capabilities, though gains vary by model and metric.

5.1. Text translation

  • Across model sizes, TranslateGemma improves MetricX over Gemma 3: scores fall from 4.04 to 3.09 (27B), 4.86 to 3.60 (12B), and 6.97 to 5.32 (4B).The reported reductions are approximately 23.5%, 25.9%, and 23.6%, respectively.Source: Table 1 · p. 4, §5.1 · p. 4
  • Comet22 also improves, including on a metric not explicitly optimized during RL: 12B rises from 81.6 to 83.5 and 4B from 77.2 to 80.1.Source: §5.1 · p. 4
  • Smaller TranslateGemma models approach or exceed larger Gemma 3 baselines: 12B surpasses baseline 27B, while 4B is comparable to baseline 12B.The report says this enables high-quality translation with reduced computational resources.Source: §5.1 · p. 4
  • TranslateGemma improves MetricX across all 55 evaluated language pairs, including high-resource and low-resource languages.Examples include English-to-German scores falling from 1.63 to 1.19 and English-to-Icelandic from 8.31 to 5.69. The authors hypothesize, without direct experimental confirmation, that broad SFT language exposure benefited the 27B model more.Source: §5.1 · p. 5

5.2. Prompting the Model

  • Evaluations used the model’s Figure 3 prompt, which the authors recommend for producing new translations.The model repository provides tools to wrap text in that prompt automatically.Source: §5.2 · p. 5

5.3. Image Translation

  • On image translation, TranslateGemma retains base-model capabilities; MetricX improves by nearly 0.5 points for 27B and 0.25 for 12B, but 12B Comet22 is an exception.The 4B model shows only small improvements, which the authors attribute probably to limited capacity. Evaluation used 264 images with a single text instance and no multimodal training data in SFT or RL.Source: Table 2 · p. 5, §5.3 · p. 5

6. Human Evaluation

Human evaluation on WMT25 translation data largely confirms TranslateGemma’s gains over Gemma 3, with notable exceptions for German-target and Japanese-to-English translation. The evaluation also finds particularly strong improvements for low-resource pairs and continued competitiveness of the 12B model.

Human Evaluation

  • Professional translators used MQM to mark translation error spans with severity and category, yielding weighted scores; annotations were collected with Anthea.The study evaluated 10 language pairs using WMT25 literary, news, and social-domain data, spanning a mix of resource levels, language families, and writing systems.Source: §6 · pp. 5–6
  • To limit rater fatigue, documents were capped at 12 source sentences or split into literary chunks, and all system outputs for each source document went to the same rater.The assignment followed the pseudo-SxS procedure of Riley et al. (2024).Source: §6 · p. 6
  • English-to-Marathi improved by 1.6 points, while English-to-Swahili and Czech-to-Ukrainian each improved by 1.0 point.Source: §6 · pp. 6–7
  • TranslateGemma outperformed Gemma 3 on most language pairs, tied when German was the target, and regressed on Japanese→English due to named-entity mistranslations despite gains in other error categories.Table 3 reports MQM results; lower scores are better.Source: §6 · p. 6, Table 3 · p. 7
  • Human evaluation confirms the performance difference between the 27B and 12B TranslateGemma models shown by automatic metrics, while 12B remains competitive with the bigger Gemma 3 model, especially for high-resource languages.Source: §6 · p. 7

7. Conclusions

TranslateGemma introduces open Gemma 3-based machine translation models trained with supervised fine-tuning and reward-model reinforcement learning. They improve translation across 55 language pairs, often let smaller models match larger baselines, and retain image-translation capabilities.

7. Conclusions

  • The open 4B, 12B, and 27B models combine supervised fine-tuning on diverse human and synthetic parallel data with reinforcement learning using an ensemble of reward models.The models are based on Gemma 3 and were developed specifically for machine translation.Source: §7 · p. 7
  • TranslateGemma models show consistent gains over baseline Gemma 3 in MetricX and Comet22 across 55 WMT24++ language pairs.The reported gains span high-resource languages such as German and Spanish and lower-resource languages such as Icelandic and Swahili.Source: §7 · p. 7
  • Smaller TranslateGemma models often match or exceed larger Gemma 3 baselines, improving the trade-off between translation quality and computational cost.Source: §7 · p. 7
  • TranslateGemma retains Gemma 3’s multimodal capabilities, and Vistra experiments show image-translation improvements, particularly for 12B and 27B models, without multimodal fine-tuning.Source: §7 · p. 7
  • The release provides open-source machine-translation tools intended to support further research and application development.Source: §7 · p. 7

Contributions

The contribution roster identifies core contributors, leads, and support.

Core Contributors

  • Mara Finkelstein, Isaac Caswell, Tobias Domhan, Jan-Thorsten Peter, Juraj Juraska, Parker Riley, Daniel Deutsch, and Geza Kovacs are listed as core contributors.Source: Core Contributors · p. 9

Lead

  • David Vilar and Markus Freitag are listed as leads.Source: Lead · p. 9

Support

  • Erin Kats and Stephane Jaskiewicz are listed under support.Source: Support · p. 9

A. Automatic metrics per language

This section compares TranslateGemma (GT) with baseline Gemma (G3) models across WMT24++ language pairs using MetricX.

A. Automatic metrics per language

  • Table 4 compares TranslateGemma (GT) and baseline Gemma (G3) performance for each WMT24++ language pair using MetricX.The supplied passage gives the comparison scope but no metric values or winning model.Source: Table 4 · p. 10

B. Additional synthetic data languages · C. Full list of languages for SFT

TranslateGemma’s SFT data included synthetic English-language pairs beyond those covered by WMT24++, while its full language coverage is organized into three tables by pairing direction and whether English is involved.

B. Additional synthetic data languages

C. Full list of languages for SFT

Related papers

Loading 2601.09012v3…