PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Compact Metrics for MT

Amy Pu, Hyung Won Chung, Ankur P. Parikh, Sebastian Gehrmann, Thibault Sellam

arXiv:2110.06341v1cs.CL

TL;DR

Learned multilingual metrics can outperform surface-overlap metrics, but strong cross-lingual transfer requires model capacity that makes evaluation costly. This paper tests the multilinguality–capacity trade-off with RemBERT and uses synthetic multilingual data to distill a teacher into students targeted to related languages. The combined approach improves over fine-tuning and gives a smaller student a substantial share of the teacher’s performance.

  • Problem

    Cross-lingual transfer gains in learned metrics depend on sufficient model capacity, while scaling models for evaluation can be costly and impractical.

    Source: §1 · p. 1

  • Method

    The approach generates synthetic multilingual sentence pairs for teacher-supervised distillation and trains multiple students focused on smaller sets of related languages.

    Source: §3 · p. 4

  • Results

    Up to 10.5% improvement compared with fine-tuning; RemBERT-12 matches 92.6% of the teacher’s performance with one-third its parameters.

    Source: §3 · p. 5

  • Takeaways & Limitations

    Distillation addresses the tension between the large multilingual models encouraged by transfer and the smaller models evaluation’s limited hardware resources require.

    Source: §2 · p. 3

    The study reports no Inuktitut results because RemBERT does not cover its alphabet.

    Source: §2 · p. 2

Abstract

from arXiv · show

Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.

1 Introduction

Learned multilingual metrics can capture semantic similarity beyond surface-level overlap, but their accuracy gains depend on model capacity, making evaluation costly. The paper tests this trade-off with RemBERT and uses synthetic data and distillation to train compact multilingual students.

1 Introduction

  • N-gram metrics such as BLEU and ROUGE focus on surface form and correlate poorly with human evaluation, especially for high-quality generated text.Learned metrics instead treat evaluation as regression and use pretrained multilingual representations.Source: §1 · p. 1
  • Large pretrained models support cross-task transfer when task data are scarce and cross-lingual transfer where few evaluation datasets cover more than 20 languages.Evidence cited in the introduction suggests multilingual training can help languages with little data, including zero-shot languages (Conneau and Lample, 2019; Sellam et al., 2020b; Conneau et al., 2018; Pires et al., 2019).Source: §1 · p. 1
  • RemBERT experiments confirm that cross-lingual transfer gains in metric evaluation depend on model capacity, motivating a compact alternative.The introduction frames large models as costly to run alongside MT or NLG systems and a barrier to researchers with limited resources.Source: §1 · pp. 1–2
  • Using an 80M-sentence, 13-language Wikipedia distillation corpus, multilingual data generation and distillation improve a vanilla pre-trained distillation setup by up to 12%; combined methods match 92.6% of the largest RemBERT model’s performance with one-third its parameters.Source: §1 · p. 2

2 Multilinguality and Model Size

Experiments on WMT Metrics data examine the relationship between multilinguality and model capacity. Larger models and multilingual fine-tuning improve performance, while reducing pretraining languages benefits the smaller models; distillation is presented as a way to balance performance and compactness.

2 Multilinguality and Model Size

  • WMT Metrics evaluates MT quality for 18 language pairs against human ratings; training uses 479k triplets in 12 languages and leaves Japanese, Polish, and Tamil zero-shot.Inuktitut results are omitted because RemBERT does not cover its alphabet.Source: §2 · p. 2
  • The metric regresses from a vector representation of a reference and an MT output to a real-valued human rating, using RemBERT as the pretrained encoder.RemBERT is described as pretrained on 104 languages using Wikipedia and mC4; the study also evaluates smaller variants with 3, 6, and 12 layers.Source: §2 · pp. 2–3
  • RemBERT-32 is on par with BLEURT-EXTENDED, and larger models perform better at a fixed number of languages.Figure 1 reports WMT Metrics DaRR, with higher values better; each result is a five-seed mean with Normal-based 95% confidence intervals.Source: Figure 1 · p. 2, §2 · p. 3
  • Adding fine-tuning languages improves zero-shot performance in all tested cases, with a milder effect for RemBERT-6, which starts higher but finishes lower.The comparison uses RemBERT-6 and RemBERT-32 on en-ja, en-pl, and en-ta without fine-tuning on the target languages.Source: §2 · p. 3, Figure 2 · p. 3
  • Reducing pretraining from 104 languages to 18 increases performance in all cases, especially for RemBERT-3.The smaller models were pretrained from scratch on 18 WMT languages and fine-tuned on the full dataset; the result suggests that the models are at capacity and languages compete in pretraining.Source: §2 · p. 3, Figure 3 · p. 3
  • Learned metrics face a trade-off: multilingual pretraining and cross-lingual transfer favor large models, while evaluation’s limited hardware resources call for smaller ones.The paper addresses this conflict with distillation.Source: §2 · p. 3

3 Addressing the Capacity Bottleneck

The section uses distillation to address the capacity demands of multilingual learned metrics, combining synthetic data and specialized students. The combined approach improves on vanilla fine-tuning while reducing model size, and also improves runtime.

3 Addressing the Capacity Bottleneck

  • Up to 10.5% improvement over vanilla fine-tuning; RemBERT-12 matches 92.6% of teacher performance with one-third its parameters.The table reports segment-level agreement with human ratings using WMT Metrics DaRR, where higher is better.Source: Table 1 · p. 4, §3 · p. 5
  • Pre-trained distillation fine-tunes RemBERT-32 on human ratings, then uses its predictions on unlabeled sentence pairs to supervise smaller RemBERT students.The default setup uses WMT sentence pairs for both teacher fine-tuning and student distillation, but with different labels.Source: §3 · pp. 3–4
  • Synthetic-data generation creates 80 million sentence pairs in 13 languages by perturbing Wikipedia sentences with back-translation, substitutions, and deletions.The perturbations are intended to expose students to surface-level noise and paraphrases that MT systems could introduce.Source: §3 · p. 4
  • 1-to-N distillation trains multiple language-focused students, each covering clusters of 3 to 18 languages, to ease the constraint on a single model.The clusters were formed around geographic proximity or linguistic relatedness, with each cluster covering at least one WMT language.Source: §3 · p. 4
  • 2.5–3X faster processing than RemBERT-32 is achieved by the final model; length-based batching and RemBERT-12 contribute successive speedups.The final model processes 4.8 tuples per second without a GPU and 86 with a GPU.Source: §3 · p. 5, Figure 5 · p. 5

4 Conclusion

The conclusion frames learned MT metrics as balancing multilinguality with model capacity, then identifies distillation as the paper’s response. It also points to broader task coverage, complementary compression methods, and increased linguistic coverage as future directions.

4 Conclusion

  • The paper identifies a trade-off between multilinguality and model capacity in learned metrics and addresses it with distillation on synthetic data.The conclusion says the work exposed this trade-off and addressed the problem with distillation.Source: §4 · p. 5
  • Future work includes generalizing the approach to other tasks and testing complementary compression methods such as pruning and quantization.Source: §4 · p. 5
  • The authors also identify increasing linguistic coverage as a direction for further work.Source: §4 · p. 5

A.1 RemBERT Pre-Training

RemBERT uses an encoder-only architecture that reallocates parameters from input embeddings to wider and deeper Transformer layers. The appendix describes the pre-training setup for the models and the differing sequence lengths used in fine-tuning.

A.1 RemBERT Pre-Training

  • RemBERT is an encoder-only BERT-like architecture that reinvests savings from reduced input embeddings in wider and deeper Transformer layers.Its input and output embeddings are decoupled during pre-training.Source: §A.1 · p. 7
  • The four RemBERT models are described by architecture and parameter count; the smaller models were trained with a modified public BERT codebase.The models were pre-trained on 104 languages using masked language modeling, with Wikipedia only for the custom models.Source: §A.1 · p. 7, Table 2 · p. 8
  • The custom RemBERT models were trained for 217 steps with Adam, a 0.0002 learning rate using 10,000 linear warm-up then inverse square-root decay, batch size 512, and 16 TPU v3 chips.To reduce model size further, they used a smaller SentencePiece model with 120K tokens.Source: §A.1 · p. 7
  • Fine-tuning used sequence length 128 for RemBERT-32 and 512 for the student models.Source: §A.1 · p. 8

A.2 Fine-Tuning for the WMT Metrics Shared Task

Fine-tuning uses WMT Metrics data from 2015 to 2019, with a held-out evaluation portion and precautions against reference leakage. The smaller models use a separate training schedule and learning rate selected by a held-out-sample sweep.

A.2 Fine-Tuning for the WMT Metrics Shared Task

  • The fine-tuning data combines WMT Metrics sentence pairs from 2015 to 2019, setting aside 5% for continuous evaluation and preventing reference translations from leaking across datasets.The passage says sentences were sampled randomly before adjusting the sample to avoid leakage.Source: §A.2 · p. 8
  • For the large model, a sweep on prior-year data selected learning rate 1e-6 from eight tested values.The sweep trained on 2015–2018 and tested on 2019.Source: §A.2 · p. 8
  • RemBERT-3, -6, and -12 used learning rate 1e-5, 20,000 training steps, and batch size 32, with evaluation every 1,000 steps.The learning rate came from a sweep on a randomly held-out sample.Source: §A.2 · p. 8

B Additional Ablation Experiments on WMT Metrics Shared task 2020

The ablations show that adding fine-tuning languages has mixed effects on trained-language pairs but usually helps zero-shot languages. For smaller models, removing most pre-training languages often improves performance, indicating a capacity limit.

B Additional Ablation Experiments on WMT Metrics Shared task 2020

  • Fine-tuning languages are added in decreasing frequency order, in buckets containing about the same number of examples.Figure 7 evaluates BLEURT across all models and WMT’20 test language pairs except en-iu, using WMT Metrics DaRR; higher is better.Source: B Additional Ablation Experiments on… · p. 8, Figure 7 · p. 9
  • Adding fine-tuning data for a language pair improves performance on that pair, while cross-lingual transfer is mixed: mild to negative on *-en and mostly positive on en-cs.The experiments start from the five languages with training data.Source: B Additional Ablation Experiments on… · p. 8
  • Adding data improves performance in almost all zero-shot cases, but the effect is milder on smaller models, with slight drops for RemBERT-3 on en-ta and en-ja.The authors describe these drops as consistent with the curse of multilinguality.Source: B Additional Ablation Experiments on… · p. 8
  • 21 of 24 cases improved after removing 86 pre-training languages, regardless of whether the test language was zero-shot.The passage describes this as further evidence that the smaller models are saturated.Source: B Additional Ablation Experiments on… · pp. 8–9, Figure 8 · p. 9

C.1 Distillation Data Generation Method

The method creates synthetic reference–MT output pairs by perturbing Wikipedia sentences, then uses teacher-generated data for distillation across language clusters. The perturbations emulate noise from MT systems, and the clusters combine linguistic proximity with geographic proximity.

C.1 Distillation Data Generation Method

  • Synthetic sentence pairs are generated by perturbing Wikipedia sentences to produce pseudo-translations and expose students to MT-like noise and paraphrases.The three perturbations are back-translation, mBERT word substitutions, and random deletions.Source: §C.1 · p. 9
  • The generation covers 13 languages chosen because they appear in WMT Metrics training, testing, or both.The languages are English, Japanese, Lithuanian, Czech, Tamil, Chinese, Russian, Kazakh, Gujarati, Finnish, French, Polish, and German.Source: §C.1 · p. 10
  • Word substitution masks up to 15 WordPiece tokens and replaces them with mBERT beam-search outputs using beam size 8.Source: §C.1 · p. 10

C.2 Languages Used in 1-to-N Distillation

  • The five student clusters first group linguistically proximate languages, then use geographic proximity to place remaining languages.For example, Tamil joins an otherwise Indo-Iranian cluster, while Japanese joins a Sino-Tibetan cluster.Source: §C.2 · p. 10, Table 3 · p. 10

C.3 Setup and Hyper-parameters

Distillation training uses 500,000 batches of 128 examples, learning from 64M sentences rather than the 640K used in fine-tuning, and runs to completion without early stopping.

C.3 Setup and Hyper-parameters

  • Distillation trains for 500,000 batches of 128 examples, or 64M sentences, and runs to completion without early stopping.Training takes about 1.5 days for RemBERT-3 and RemBERT-6, and 3.5 days for RemBERT-12.Source: §C.3 · p. 10

D Additional Details of Metrics Performance

The benchmark analysis reports distinct outcomes across segment- and system-level correlations. Distillation brings solid improvements on from-English system-level results, while its impact on to-English system-level results is mixed.

D Additional Details of Metrics Performance

  • The re-implemented benchmark results are consistent with the published version except for segment-level to-English pairs marked with a dagger.The reported performance covers system- and segment-level agreement on the WMT Metrics Shared Task 2020.Source: D Additional Details of Metrics… · p. 10, Table 4 · p. 11, Table 5 · p. 11
  • RemBERT-32 is outperformed by several other metrics on system-level en-* and *-en results.The passage contrasts these system-level outcomes with segment-level results and notes mixed distillation effects on to-English.Source: D Additional Details of Metrics… · p. 11
  • Up to 20.2% improvement is reported on from-English system-level results, despite mixed distillation effects on to-English system-level results.The authors note that system- and segment-level correlations can yield very different outcomes.Source: D Additional Details of Metrics… · pp. 11–12, Table 7 · p. 13

Related papers

Loading 2110.06341v1…