Source-linked AI summary
Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality
Carolina Scarton, Mikel L. Forcada, Miquel Esplà-Gomis, Lucia Specia
TL;DR
Because MT evaluation depends on translation purpose, this paper compares ways to estimate quality for post-editing. Using newly collected post-editing indicators, it finds task-specific metrics track post-editing effort best, followed by DA and independent-reference metrics.
Problem
Because MT assessment depends on translation purpose, the paper examines how to estimate MT quality for post-editing, where effort matters to productivity.
Source: §1 · pp. 1–2Method
The authors release a dataset with post-edited texts, five independent post-editing jobs per MT text, and DA annotations, then compare metric rankings and introduce SATRA.
Source: §1 · p. 2, §3.1 · p. 3Results
Task-specific metrics best track PETpW, with Keys/char best overall; DA follows, and independent-reference metrics rank worst.
Source: §3.1 · p. 4, §5 · p. 6Takeaways & Limitations
MT practitioners should account for these metric differences when choosing how to evaluate MT for post-editing.
Source: §5 · p. 6With only five annotators, it is difficult to devise a model that estimates quality well for new annotators.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Devising metrics to assess translation quality has always been at the core of machine translation (MT) research. Traditional automatic reference-based metrics, such as BLEU, have shown correlations with human judgements of adequacy and fluency and have been paramount for the advancement of MT system development. Crowd-sourcing has popularised and enabled the scalability of metrics based on human judgements, such as subjective direct assessments (DA) of adequacy, that are believed to be more reliable than reference-based automatic metrics. Finally, task-based measurements, such as post-editing time, are expected to provide a more detailed evaluation of the usefulness of translations for a specific task. Therefore, while DA averages adequacy judgements to obtain an appraisal of (perceived) quality independently of the task, and reference-based automatic metrics try to objectively estimate quality also in a task-independent way, task-based metrics are measurements obtained either during or after performing a specific task. In this paper we argue that, although expensive, task-based measurements are the most reliable when estimating MT quality in a specific task; in our case, this task is post-editing. To that end, we report experiments on a dataset with newly-collected post-editing indicators and show their usefulness when estimating post-editing effort. Our results show that task-based metrics comparing machine-translated and post-edited versions are the best at tracking post-editing effort, as expected. These metrics are followed by DA, and then by metrics comparing the machine-translated version and independent references. We suggest that MT practitioners should be aware of these differences and acknowledge their implications when deciding how to evaluate MT for post-editing purposes.
1. Introduction
MT quality depends on its intended use, and post-editing usefulness therefore calls for evaluation against the effort of that task. The paper reviews and contributes measures for ranking machine-translated segments by post-editing effort.
1. Introduction
- An MT sentence can mislead readers yet need only one word corrected, while an understandable sentence with multiple errors can require at least five word-level edits.The contrast illustrates why end-user comprehensibility and post-editing effort are not interchangeable measures of usefulness.Source: §1 · p. 1
- For dissemination, machine-translated text is intended for publication, whereas assimilation prioritizes communication and can tolerate grammatical errors if the message is understood.The paper focuses on post-editing for dissemination, while noting that heavily problematic sentences may be easier to translate from scratch.Source: §1 · p. 1
- Post-editing time per word (PETpW) measures temporal effort, but collecting it is relatively expensive and requires uninterrupted editing of each sentence.The paper normalizes time by the machine-translated segment’s word count because time is expected to grow roughly linearly with sentence length.Source: §1 · p. 2
- The paper reviews task-specific, reference-based, and direct-assessment metrics for ranking segments by post-editing time, and releases a dataset with five independent post-editing jobs per segment.The dataset also includes source, machine-translated, reference, and post-edited texts, plus direct-assessment annotations.Source: §1 · p. 2
- The paper introduces split-averaged, time-ratio assessment (SATRA), a new ranking score for machine-translated segments.Estimating post-editing effort is also framed as useful for translation-job budgeting.Source: §1 · p. 2
2. Dataset and annotators
The dataset extends an existing English–Spanish quality-estimation collection with post-editing work by five professional translators. It combines task-specific and reference-based indicators with direct assessments.
2. Dataset and annotators
- The study extends a WMT 2016 dataset of 1,047 segments and 26,875 words, machine-translated by 41 systems from English–Spanish shared-task test sets.Existing segments received crowd-annotated direct-assessment scores.Source: §2 · p. 2
- Five professional translators with post-editing experience produced edited versions for every segment using PET, which records edit operations and elapsed post-editing time.The study uses post-editing time normalized by the machine-translated segment’s word count.Source: §2 · p. 3
- The evaluation includes HTER, HBLEU, HMETEOR, and keys per character as task-specific indicators, alongside reference-based BLEU, METEOR, and TER.The task-specific scores compare machine-translated text with post-edited text; keys per character is the number of keys pressed divided by the segment’s character count.Source: §2 · p. 3
- Metric statistics are reported by annotator and overall for task-specific indicators, and for direct assessments and reference-based metrics; averages are weighted by machine-translated word count.The authors motivate weighting by the expectation that post-editing time grows linearly with segment length, and report a wide range of quality-indicator values.Source: §2 · p. 3, Table 2 · p. 3, Table 3 · p. 3
3. Comparing the ranking ability of metrics
The section compares metric rankings with post-editing time using Spearman correlation, SATRA, and analyses of the ranking tails. Across these analyses, PE-based metrics generally track PETpW best, DA falls between them and independent-reference metrics, and Keys/char leads the full-ranking comparison.
3.1. Ranking correlation
- SATRA complements Spearman’s ρ by comparing average PET time per word above and below ranking splits; values near 1 indicate a random ranking, while lower values indicate better rankings.The score is lowest for a ranking based on measured PETpW.Source: §3.1 · p. 3
- Keys/char is best overall across individual annotators and ALL, while independent-reference metrics rank worst against PETpW; PE-based metrics outperform DA in most cases.DA has lower Spearman correlations and worse SATRA scores than the task-specific PE-based metrics.Source: §3.1 · pp. 3–4, Table 4 · p. 4
- In leave-one-out tests, PE-based metrics generally retain better SATRA scores than DA, but Keys/char is no longer best overall and leads SATRA in three of five cases.The test compares each annotator’s metric scores with the average PETpW of the other annotators.Source: §3.1 · p. 4, Table 5 · p. 4
- With only five annotators, the authors identify three PETpW distribution clusters and note that clustering may affect results; deeper analysis is left for future work.ANN0, ANN2, and ANN4 cluster together, while ANN1 and ANN3 each form their own cluster.Source: §3.1 · p. 4
3.2. Analysis of tails
- Among the 500 easiest segments, HTER, HBLEU, HMETEOR, and Keys/char share the most PETpW-ranked segments; DA falls between them and reference-based metrics.BLEU, TER, and METEOR share the fewest segments with the PETpW ranking.Source: §3.2 · p. 5, Figure 1 · p. 5
- Across all segments, DA and HTER look similar against PETpW, but DA has more outliers; among the easiest 500, HTER clusters at low PETpW while DA is sparser.The scatter plots average results over all five annotators.Source: §3.2 · p. 5, Figure 2 · p. 5
- For the 500 hardest segments, the same broad ranking pattern holds, but the gap between DA and task-specific PE metrics is smaller.The authors hypothesize that adequacy differences may track PE-time differences better among very low-quality segments.Source: §3.2 · p. 5, Figure 3 · p. 6
4. Related work
Related work assesses post-editing effort through task measures, perceived judgments, eye-tracking, and edit-based proxies. Prior work also debates whether direct assessment (DA) is a useful proxy for PE effort.
4. Related work
- PE time treats segments requiring longer edits as worse, while PETpW can measure productivity gains over translating from scratch.Koponen et al. relate PE time to cognitive aspects of PE; Plitt and Masselot use words per hour in real workflows.Source: §4 · p. 6
- Perceived PE effort uses Likert-scale judgments of how difficult a segment would be or was to fix, with or without actual post-editing.These scores were used in the WMT 2012 and WMT 2014 QE shared tasks.Source: §4 · p. 6
- Eye-tracking work associates lower GTM scores with longer fixation time, while long keystroke-log pauses are associated with greater cognitive PE effort.O’Brien measures fixation time and correlates it with GTM, a translation-reference similarity metric based on precision, recall, and F-measure.Source: §4 · p. 6
- Edit-based proxies include HTER, the minimum edits needed to transform MT into post-edited text, and related variants HBLEU and HMETEOR.HTER is described as the most widely used indirect measure of PE effort.Source: §4 · p. 6
- Although prior studies favor DA over HTER using DA correlations, this analysis finds PE-based metrics track PETpW better than DA.The authors note that comparing mTER and HTER against PETpW could yield different results, leaving that analysis for future work.Source: §4 · p. 6
5. Concluding remarks
For dissemination through post-editing, the paper compares DA and reference-based metrics with measurements collected during post-editing. Task-specific metrics track PE time better than DA, which in turn outperforms independent-reference metrics.
5. Concluding remarks
- Task-specific comparisons of MT and post-edited text, including HTER and keystrokes per raw MT character, track PETpW better than DA; DA still reasonably approximates PE time.The experiments use PE indicators collected for five translators.Source: §5 · p. 6
- DA performs better than BLEU, TER, or METEOR when those metrics use an independent reference translation.Source: §5 · p. 6
- The authors recommend task-specific metrics when feasible, and non-expert DA only when task-specific measurable metrics are unavailable or impractical.Source: §5 · p. 6