PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, André F. T. Martins

arXiv:2309.11925v1cs.CL

TL;DR

Quality estimation scores machine-translation output without reference translations, and the shared task included language pairs absent from training data. The Unbabel–IST team built multilingual systems on COMETKIWI-22 and ranked first across all tasks, surpassing that model by up to 10 Spearman correlation points.

  • Problem

    Quality estimation must score machine-translation output without reference translations, while some shared-task test language pairs were absent from training data.

    Source: §1 · p. 1

  • Method

    The team built generic QE models from multilingual annotated data, fine-tuned them for the subtasks, and built on COMETKIWI-22.

    Source: §1 · p. 1, §3 · p. 2, §5 · p. 6

  • Results

    Their multilingual systems ranked first across sentence-, word-, and span-level tasks and surpassed COMETKIWI-22 by up to 10 Spearman correlation points.

    Source: §1 · p. 1, §5 · p. 6

  • Takeaways & Limitations

    The team publicly released two of its best models, described as the largest open-source QE models publicly released to the authors’ knowledge.

    Source: §1 · p. 1

    GPT-4 QE is expensive and slow, and its error-flagging precision and recall vary with prompt examples, their ordering, and the number of errors per example.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

We present the joint contribution of Unbabel and Instituto Superior T\'ecnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2). For all tasks, we build on the COMETKIWI-22 model (Rei et al., 2022b). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art COMETKIWI-22, we show large improvements in correlation with human judgements (up to 10 Spearman points). Moreover, we surpass the second-best multilingual submission to the shared-task with up to 3.8 absolute points.

1 Introduction

The Unbabel–IST team develops multilingual quality-estimation systems for all WMT23 QE tasks, aiming to generalize to language pairs absent from training. Its systems rank first across sentence-, word-, and span-level tasks.

1 Introduction

  • To address test language pairs absent from training, the team built generic QE models from nearly 1M annotations across 38 language pairs and fine-tuned them for the subtasks.The corpus combined WMT Metrics direct assessments from 2017–2020, MLQE-PE, and 2023 shared-task annotations for Indian languages.Source: §1 · p. 1
  • Sentence-level systems were fine-tuned on this year’s direct-assessment data using XLM-R XL or XXL, while word-level training combined sentence- and word-level signals in one loss.For span detection, the team explored approaches based on its word- and sentence-level strategies, and contrasted them with UnbabelQi and GPT-4.Source: §1 · p. 1
  • The team introduced multilingual approaches ranked first at word, span, and sentence granularity, explored error-span severity prediction, and publicly released two COMETKIWI models.The released COMETKIWI-XL and -XXL models are described as the largest open-source QE models publicly released, to the authors’ knowledge.Source: §1 · p. 1
  • 59.4 Spearman correlation points, 31.7 MCC score, and 22 F1.0 score were achieved for sentence-, word-, and span-level prediction, respectively.The results surpassed the second-best system by nearly 4, almost 2, and more than 5 absolute points, respectively.Source: §1 · p. 2

2 Overview of the shared-task

The shared task covered sentence- and word-level quality prediction and fine-grained error-span detection. Submissions were evaluated on different annotation types and language-pair sets across these tasks.

2 Overview of the shared-task

  • Sentence-level QE predicts one score from a source and translation, word-level QE labels each translated word OK or BAD, and span detection identifies erroneous character sequences and their severity.The shared-task severity labels were MINOR and MAJOR, with non-error portions sometimes labelled OK.Source: §2 · p. 2
  • Sentence-level submissions were evaluated against direct assessments for five language pairs and MQM annotations for three; training data was unavailable for he-en.The direct-assessment pairs were en-mr, en-hi, en-ta, en-te, and en-gu; the MQM pairs were en-de, zh-en, and he-en.Source: §2 · p. 2
  • Word-level submissions used post-edition-inferred tags for two language pairs and MQM annotations for three, with no additional word-tag training or development data available.The authors knew of no word-level data for en-fa or he-en.Source: §2 · p. 2
  • Span-detection submissions were evaluated on MQM-derived error spans for en-de, zh-en, and he-en, but no training or development data was available for he-en.Source: §2 · p. 2

3 Implemented Systems

The systems build on COMETKIWI, combining multilingual encoders and task-specific adaptation for sentence-, word-, and span-level quality estimation. Training data, model outputs, and prediction strategies vary across the shared-task settings.

3 Implemented Systems

  • The COMETKIWI architecture combines source and translation representations through a learned weighted sum of encoder-layer hidden states, then uses task-specific prediction heads.Sentence-level prediction uses the <cls> representation and a two-layer feedforward module; word-level and span models project token representations.Source: §3 · pp. 2–3
  • Generic models are trained on over 940k sentence-level quality samples spanning 38 language pairs, then adapted to each shared-task stream.Training uses WMT 2017–2019 DA annotations, MLQE-PE, and this year’s annotations for seven language pairs.Source: §3 · p. 3

3.1.1 Sentence-level quality prediction

  • For sentence-level QE, DA annotations from the current shared task further fine-tune generic models, while MQM language pairs retain the generic models after adaptation attempts hurt DA correlations elsewhere.The final ensemble combines two XL and two XXL models, with weights selected separately for each language pair.Source: §3.1.1 · pp. 3–4

3.1.2 Word-level quality prediction

  • Word-level models use prior training data because this year provided no word-tag training or development data, and the final ensemble combines six models with optimized weights.The models omit language prefixes for zero-shot evaluation; five use InfoXLM and one uses XLM-R XL, which did not yield clear performance boosts when scaled up.Source: §3.1.2 · p. 4

3.2 Task 2: Fine-grained error span detection

  • For span detection, consecutive BAD tags can become character spans, with all span subwords assigned MINOR or MAJOR severity; the latter performed best among these options.A second approach uses XCOMET with DeepL or Google pseudo-references, first scoring reference quality and using QE-only input below 0.5; above 0.5, it weights source-only, reference-only, and unified inputs.Source: §3.2 · p. 4

4 Experimental Results

Across sentence-, word-, and span-level quality estimation, the results compare retrained and scaled models, post-edit and MQM systems, and alternative span-detection approaches. The reported span-detection leader uses an ensemble of larger models and a pseudo-reference, while GPT-4 sentence scoring has stated limitations.

4 Experimental Results

  • Sentence-level systems were ranked primarily by Spearman correlation, word-level systems by MCC, and span detection by F1 with severity-sensitive credit for true positives.The reported evaluations separately covered in-domain and zero-shot language pairs for direct assessments and MQM.Source: §4 · p. 5

4.1 Quality Estimation

  • The retrained InfoXLM system improved overall by 4.1 Spearman points, while maintaining en-de correlations and showing a drop for zh-en.The en-de comparison involved a high-resource pair; he-en had not been seen during training.Source: §4.1 · p. 5
  • Among the three backbones, XXL improved across language pairs over InfoXLM; further fine-tuning improved Indian-language results and preserved or sometimes raised MQM performance.The ensemble of high-performing models was the best submission.Source: §4.1 · p. 5
  • For word-level prediction, post-edit models were competitive on MQM pairs, and the final ensemble relied more heavily on post-edit models except for en-de.The best he-en result came from a post-edit word-level model; the en-zh and en-he ensembles used only post-edit models.Source: §4.1 · pp. 5–6
  • xCOMET-PS-REF surpassed both the word-level model and UNBABELQI for error-span detection; the authors attribute its improvement to larger models and a pseudo-reference.The pseudo-reference was particularly beneficial on he-en, for which no training data was available.Source: §4.1 · p. 6
  • GPT4-QE led average F1 mostly through en-de, but span-derived sentence scores correlated poorly with DA judgments and low-resource he-en, and its precision and recall were hard to control.The authors also report that prompt composition changed error-flagging propensity, and GPT-4 was expensive and slow even for the shared task.Source: §4.1 · p. 6

5 Final Remarks

The final systems rank first across sentence-, word-, and span-level multilingual quality estimation, improving on COMETKIWI-22 by up to 10 Spearman points. They retain the previous architecture while using more data, larger encoders, and complementary model ensembles.

5 Final Remarks

  • The multilingual systems rank first across all three translation-quality granularities and surpass COMETKIWI-22 by up to 10 Spearman correlation points.Source: §5 · p. 6
  • The systems follow last year’s COMETKIWI architecture while leveraging more data and larger encoder models.Source: §5 · p. 6
  • The best final systems ensemble models trained on DA, post-edits, or MQM scores, which the authors describe as complementary.Source: §5 · p. 6
  • For sentence-level translation quality prediction, the best systems surpass GPT-4 by a large margin and are comparable to GPT-4 at error span detection.Source: §5 · p. 6

Related papers

Loading 2309.11925v1…