MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

1 Introduction

• MetricX-24 is a learned regression metric trained to predict a floating-point translation-quality score.The submissions are based on mT5, fine-tuned on DA and MQM ratings and newly constructed synthetic data.Source: §1 · p. 1
• MetricX-24-Hybrid is one model that can score a translation with the source, the reference, or both provided as input.The same model serves as the primary submission for both reference-based evaluation and quality estimation.Source: §1 · p. 1
• The paper reports that learned metrics miss undertranslation, duplication, missing punctuation, and fluent but unrelated translations, while synthetic training data improves performance especially on such lower-quality translations.It also reports that mixing MQM and DA ratings maintains high performance across more language pairs, and that hybrid-input training lets the metric rely less on poor-quality references.Source: §1 · p. 1

2 Data

• MQM ratings are rubric-based and annotated by professionals but cover few language pairs, whereas DA ratings cover nearly 50 pairs but are noisier and generally lower quality.The paper notes that per-rater z-normalization can make DA ratings more comparable, while MQM ratings do not benefit from z-normalization.Source: §2 · p. 2

2.1 Training Data

• Training uses DA ratings from WMT15–22 with exclusions, plus raw MQM ratings from 2021–22 for en-de, en-ru, en-zh, and zh-en.The excluded DA data include WMT21 into-English pairs, WMT22 into-English pairs, and WMT22 en-zh; the en-zh MQM training data are limited to three domains.Source: §2.1 · p. 2

2.2 Evaluation Data

• Evaluation uses WMT23 MQM ratings and WMT22 en-zh news ratings, with WMT23 DA ratings serving as a secondary set for experiments mixing DA and MQM training data.The WMT23 MQM set includes en-de, he-en, and zh-en; zh-en references are replaced because the originals are known to be low quality, while its source-based MQM ratings are unchanged.Source: §2.2 · p. 2

2.3 Synthetic Data

• The new synthetic examples target translation failures that are underrepresented in regular WMT training data and not covered by the DEMETR challenge set used previously.The examples are created by modifying DA data from WMT15–22, spanning 49 language pairs.Source: §2.3 · pp. 2–3, §2.3.1 · p. 3

2.3.1 Training Sets

• Synthetic training examples are generated by modifying WMT DA examples to represent failure modes the metric may not recognize.The paper describes the failure-mode categories and example-generation approach in Table 1.Source: §2.3.1 · p. 3

2.3.2 Test Set

• The synthetic test set pairs each generated example with its original counterpart so metrics can be evaluated on how often they rank the pair correctly.The examples are based on WMT23 DA data and are generated analogously to the synthetic training examples.Source: §2.3.2 · p. 3

3 Metric Descriptions

• MetricX-24 submissions build on MetricX-23 and share several improvements before diverging into individual variants.The section outlines shared properties, modifications, and differences among submissions.Source: §3 · p. 3

3.1 MetricX Model

• MetricX-24 uses mT5 and predicts a translation-quality score, with two-stage fine-tuning first on DA ratings and then on raw MQM ratings.The DA targets are z-normalized, aggregated per segment, negated, and clipped to [−1.0, 1.0]; the output scores are in [0, 25].Source: §3.1 · p. 3

3.2 Design Improvements

• Adding WMT22 DA and MQM ratings initially improved performance, and the design changes also add broader synthetic data, mixed DA/MQM training, and hybrid inputs.The additional MQM ratings include en-ru, a language pair absent from older MQM data, and are valuable given MQM data scarcity.Source: §3.2 · p. 3

3.2.1 Training With Synthetic Data

• Synthetic examples cover more failure modes than the empty and reference-matching cases used for MetricX-23 and are mixed with DA and MQM data in both fine-tuning stages.The selected synthetic-example ratios were 1:100 per category in stage one and 1:5000 in stage two.Source: §3.2.1 · pp. 3–4

3.2.2 Mixing DA and MQM Data

• Mixing DA ratings into the MQM-focused second stage is intended to retain performance across the broader set of language pairs represented in DA data.Raw MQM ratings combined with raw DA ratings linearly transformed to the MQM [0, 25] scale performed best; a 1:4 DA:MQM ratio restored DA-set performance to first-stage levels without significant negative impact on MQM evaluation.Source: §3.2.2 · p. 4

3.2.3 Hybrid Input Mode

• Training on source+hypothesis, hypothesis+reference, and source+hypothesis+reference inputs lets the hybrid model operate in QE and reference-based modes.The paper says this training gives the model an opportunity to adjust source and reference weights, including potentially ignoring a low-quality reference.Source: §3.2.3 · p. 4

3.3 MetricX-24 Variants

• The primary reference-based and QE submissions use the same hybrid model, while the secondary counterparts are trained only with references included or omitted, respectively.The variants otherwise share training-data mixtures and hyperparameters.Source: §3.3 · p. 5

4.1 Meta-Evaluation

• The primary evaluation uses WMT23 MQM ratings and WMT22 en-zh news ratings, focusing on en-de and en-zh while retaining zh-en to check against overfitting to out-of-English pairs.Model scores are compared with human judgments using four correlations from the WMT23 Metrics Shared Task.Source: §4.1 · p. 5

4.2 Checkpoint Selection

• Checkpoint selection combines segment- and system-level pairwise accuracy, downweighting the system-level component because of its greater variance.The downweighting is intended to avoid selecting a checkpoint based on a rare system-level accuracy spike when segment-level accuracy is low.Source: §4.2 · p. 5

4.3 Implementation Details

• All MetricX-24 variants use mT5-XXL with 13B parameters and are implemented with TensorFlow and T5X.The submitted checkpoints are released after conversion from TensorFlow to PyTorch.Source: §4.3 · p. 5

5 Results and Discussion

• Each experiment used one random seed, and the reported results focus on fully trained models and second-stage modifications.Reference-based ablations are discussed in depth; QE results are provided in Appendix D.2.Source: §5 · p. 5

5.1 Training With Synthetic Data

• Missing-punctuation training examples did not help, plausibly because imperfect references and minor punctuation penalties can yield misleading labels; reference-matching remained an exception to the broader robustness gains.The authors report MetricX-24 as more robust than MetricX-23 across failure modes overall, except for reference-matching translations.Source: §5.1 · pp. 5–6
• 94–100% accuracy was reached in most synthetic-test categories with synthetic training data, which was generally more effective than adding DA data; using both performed similarly to synthetic data alone.The dedicated duplication training set was omitted because models identified those cases without it.Source: §5.1 · p. 5, Table 2 · p. 6

5.3 Hybrid Input Mode

• MetricX-24-Hybrid gains 1–2 points in segment-level accuracy over MetricX-23 and improves en-zh system-level accuracy by 11.5 points.The paper describes these as consistent segment-level gains and a significant improvement over its previous submission.Source: Table 3 · p. 6, §5.3 · p. 7

5.2 Mixing DA and MQM Data

• Mixing DA with MQM yielded only minor MQM-evaluation changes within expected run variance, while WMT23 DA pairwise accuracy rose by up to 2 and 5 points at segment and system levels, respectively, and Pearson’s r by up to 10 points.The authors report these DA gains without sacrificing performance on the MQM evaluation set.Source: §5.2 · p. 6

5.3 Hybrid Input Mode

• At segment level, the hybrid variant outperformed the non-hybrid model across the evaluated language pairs, including almost 1% higher accuracy on en-zh; at system level, it lagged slightly in zh-en and en-zh.The other performance differences were largely insignificant.Source: §5.3 · p. 7

6 Related Work

• Metrics may predict scalar scores, structured outputs, or natural-language explanations.Scalar-score examples include BLEU, ChrF, MetricX, BLEURT, COMET, COMETKiwi, and Prism; structured and explanation-producing methods are also described.Source: §6 · p. 7
• Reference-based metrics require a reference translation, whereas reference-free metrics, also called quality estimation, do not.Metric developers commonly train separate models for the two settings.Source: §6 · p. 7
• MetricX-24 combines reference-based and reference-free evaluation in one hybrid model, rather than using separate models for each task.The paper notes prior work taking a similar single-model approach (Guerreiro et al., 2023).Source: §6 · p. 7
• Recent metrics use general-purpose LLMs, but dedicated learned models led segment-level performance in the previous year’s Metrics Shared Task.The paper contrasts this with strong system-level performance by LLM-based metrics.Source: §6 · p. 7

• All four MetricX-24 versions significantly outperform MetricX-23 on the WMT23 MQM dataset.Source: §7 · p. 7
• MetricX-24 is more robust to undertranslation and fluent but unrelated translations, which do not frequently occur in WMT data.Source: §7 · p. 7
• Combining DA and MQM ratings in final-stage fine-tuning dramatically increases performance on WMT23 DA across eight language pairs while maintaining high MQM correlations.Source: §7 · p. 7
• The four submissions include reference-based and QE variants, plus a hybrid variant evaluated with and without references.Source: §7 · p. 7

• The authors sample 500 examples from each language pair and manipulate candidate translations to create synthetic examples for each failure-mode category.Source: A Synthetic Data Creation · p. 10
• Missing-punctuation examples instead use stratified sampling across 11 end-punctuation symbols, with 250 examples each.Source: A Synthetic Data Creation · p. 10
• Synthetic examples generally worsen or corrupt the candidate, but reference-matching examples teach the metric to score reference-matching translations highly.The paper says this behavior is not learned reliably from WMT-only training.Source: A Synthetic Data Creation · p. 10

• At system level, pairwise ranking accuracy measures agreement in system-pair rankings, while Pearson’s r measures the linear relationship between metric and human scores.System scores are means of segment-level scores for each system.Source: B Meta-Evaluation Details · p. 10
• At segment level, group-by-item pairwise accuracy with tie calibration compares translations for the same source and averages accuracy across input segments.It rewards correct ties by treating metric-score differences below an automatically determined threshold as ties.Source: B Meta-Evaluation Details · p. 10
• Segment-level no-grouping Pearson’s r measures the linear relationship between metric and human scores across translations from every system and document.Source: B Meta-Evaluation Details · p. 10

C Implementation Details

• The maximum input length increases from 256 to 512 SPM tokens, with up to three segments—source, hypothesis, and reference—in each input.Source: C Implementation Details · p. 10
• Models train for 16K steps in stage one, then fine-tune the best checkpoint for 8K steps in stage two using a lower learning rate.The learning rates are 0.001 and 0.0002, respectively; decay begins after 2K steps in stage one and 1K steps in stage two.Source: C Implementation Details · p. 10
• Each training run uses 256 TPUs and batch size 256, with the Adafactor optimizer.Source: C Implementation Details · p. 10

D.1 Mixing DA and MQM Data

• The DA-mixing experiments contrast four two-stage variants with DA-only training across en-de and zh-en, present in MQM training, and en-cs and de-en, absent from it.Table 5 reports reference-based meta-evaluation scores on this subset of WMT23 DA language pairs.Source: §D.1 · p. 10, Table 5 · p. 12
• After MQM-only fine-tuning, en-cs pairwise accuracy on WMT23 DA falls from 59.54 to 57.43 at segment level and from 87.62 to 82.86 at system level.The authors note that rating-distribution differences can lower Pearson correlation but should not significantly affect rankings.Source: §D.1 · p. 10
• Adding a small proportion of DA data in the second stage restores most first-stage performance; adding synthetic data sometimes boosts it further, without sacrificing MQM or synthetic-test performance.Table 5 compares reference-based variants trained on DA only or in two stages; the MQM and synthetic-test comparisons are reported in Tables 2 and 3.Source: §D.1 · p. 12, Table 5 · p. 12

D.2 QE Models

• QE models score lower on missing-punctuation and reference-matching synthetic examples, partly because those examples are constructed from references and QE lacks reference input.For missing punctuation, the hypothesis is a reference with end punctuation removed; for reference matching, QE cannot see the reference quality.Source: §D.2 · pp. 12–13, Table 7 · p. 13
• On WMT23 MQM pairwise accuracy, the hybrid QE model does not outperform the standalone model, although most differences are within expected variance.Table 8 presents the reference-free scores; its alternate-reference zh-en set includes the original references as an additional human system.Source: §D.2 · p. 13, Table 8 · p. 13
• Both primary and secondary QE submissions outperform reference-based MetricX-23 on the WMT23 MQM evaluation set.Table 8 reports meta-evaluation scores for reference-free variants on WMT23 MQM.Source: §D.2 · p. 13, Table 8 · p. 13