Source-linked AI summary
A Set of Recommendations for Assessing Human-Machine Parity in Language Translation
Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, Antonio Toral
TL;DR
As MT quality has improved, claims of parity with professional translators have raised questions about how human evaluation is designed. Reassessing Chinese-to-English news translation, the authors find that judgments depend on raters, context, and reference translations, and recommend revisiting evaluation practices.
Problem
Prior evaluations reported human–machine parity, while human evaluation design choices may affect the validity of such judgments.
Source: §1 · pp. 1–2Method
The authors reassess Hassan et al.’s Chinese-to-English study by examining rater choice, linguistic context, and reference translations.
Source: §7 · pp. 17–18Results
Professional raters favored human translation; human translation was judged more accurate in full-document than out-of-context sentence evaluation, and reference editing could reduce adequacy.
Source: §7 · p. 18Takeaways & Limitations
The authors recommend revisiting human evaluation design to strengthen assessments of strong MT and human–machine parity, while noting the cost of professional raters and full-document evaluation.
Source: §7 · p. 18The results rely on few raters and one news domain and language direction; broader experiments are needed to increase external validity.
Source: §6 · p. 17
Abstract
from arXiv · showhide
The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We reassess Hassan et al.'s 2018 investigation into Chinese to English news translation, showing that the finding of human-machine parity was owed to weaknesses in the evaluation design - which is currently considered best practice in the field. We show that the professional human translations contained significantly fewer errors, and that perceived quality in human evaluation depends on the choice of raters, the availability of linguistic context, and the creation of reference translations. Our results call for revisiting current best practices to assess strong machine translation systems in general and human-machine parity in particular, for which we offer a set of recommendations based on our empirical findings.
1. Introduction
As MT quality has risen, evaluation design has become important to claims of human–machine parity. This paper reassesses such claims and tests how raters, context, and reference translations affect judgments, finding human translations favored when evaluation design is strengthened.
1. Introduction
- Human evaluation remains the gold standard, but its validity may depend on design choices as MT quality improves.The paper frames its investigation as a response to the challenges created by recent gains in neural MT.Source: §1 · p. 1
- Earlier evaluations reported that neural MT performed at or above professional human translation in Chinese-to-English and English-to-Czech news translation.Both used source-based direct assessment with non-expert annotators and WMT data and protocols; the authors argue that these parity findings need critical reassessment.Source: §1 · pp. 1–2
- The study tests how rater choice, linguistic context, and reference-translation quality affect human evaluation of MT, especially claims of human–machine parity.It uses the data shared by Hassan et al. (2018) and offers recommendations based on its empirical findings.Source: §1 · p. 2
2. Background
Human MT evaluation uses source texts or reference translations as rating hints, but methods differ in what they measure and who rates outputs. The paper examines parity claims by testing rater expertise, linguistic context, and reference quality, using four translation outputs.
2.1 Human Evaluation of Machine Translation
- Human evaluation is considered necessary for reliable MT quality assessment despite its high cost, and raters score or adapt outputs against a source text or reference translation.Relative ranking compares multiple outputs and permits ties, while direct assessment scores one output at a time; the former has stronger annotator agreement, but does not measure the magnitude of differences.Source: §2.1 · pp. 2–3
2.2 Assessing Human–Machine Parity
- Prior parity claims include Hassan et al.’s (2018) finding of no significant rating difference and Bojar et al.’s (2018) report that the best English-to-Czech system outperformed a human reference.The study focuses on rater choice, linguistic context, and reference construction as factors in human evaluation of parity.Source: §2.2 · p. 3
- The authors hypothesize that professional translators may distinguish strong MT from human translations better than non-experts, and that context and reference quality may affect parity judgments.They note that crowd-worker findings from older MT systems may not carry over to contemporary neural MT; sentence-level evaluation may miss context-dependent errors, and reference quality can vary.Source: Choice of Raters · pp. 3–4, Linguistic Context · p. 4, Reference Translations · p. 4
2.3 Translations
- The experiments use Hassan et al.’s professional translations (HA), a second professional translation set (HB), and two MT outputs (MT1 and MT2).HB was ordered from a different vendor and received a post-hoc native English check; MT1 was Hassan et al.’s Combo-6 system, and MT2 was Google Translate output from October 2017.Source: §2.3 · pp. 4–5
3. Choice of Raters
Expertise affected judgments of human and machine translation: experts showed a wider rating gap than non-experts. The experiment used relative rankings, while the authors note that rater performance may vary over time.
3. Choice of Raters
- Prior work found that crowd workers tended to give MT higher fluency and adequacy scores than professional translators did.The cited authors argued that non-experts may miss subtle translation differences and accept difficult MT output rather than improve it.Source: §3 · p. 5
3.1 Evaluation Protocol
- Five raters—two professional translators and three non-expert NLP researchers—ranked HA, MT1, and MT2 across 299 sentences from 41 documents.The experiment showed source sentences alongside the translations, with neighboring source sentences providing local inter-sentential context; rankings were analyzed using TrueSkill.Source: §3.1 · pp. 5–6
3.2 Results
- Expert raters showed a wider rating gap between MT1 and HA than non-experts, whose more tolerant judgments were associated with lower inter-annotator agreement.Agreement was κ = 0.13 for non-experts versus κ = 0.254 for experts.Source: §3.2 · p. 6, Table 1 · p. 6
- Rater performance may change over time through learning or fatigue, and the authors recommend accounting for this in future work, for example with trial number as a predictor.They consider such longitudinal effects likely to be present in their data.Source: §3.2 · p. 6
4. Linguistic Context
Sentence-level evaluation can miss errors involving textual cohesion and coherence, and raters may not understand a sentence’s intended meaning out of context. The section tests whether document-level context changes judgments of professional human and machine translations.
4. Linguistic Context
- Sentence-level evaluation can miss mistranslations when raters do not understand a sentence out of context and cannot assess textual cohesion or coherence.The authors note that human raters may not understand the intended meaning of an isolated sentence.Source: §4 · p. 7
4.1 Evaluation Protocol
- The evaluation compared HA and MT1 using pairwise rankings of isolated sentences and full documents from native-Chinese WMT 2017 Chinese–English news articles, without reference translations.Raters evaluated both conditions but never saw the same text in both, and item order and choice placement were randomized; spam items were used for quality control.Source: §4.1 · pp. 7–8
4.2 Results
- With document context, MT1 preference fell from 49.5 to 37.0% for adequacy and 31.7 to 22.0% for fluency; HA significantly beat MT1 on document adequacy and at both fluency levels.The results come from blind pairwise rankings by professional translators.Source: Table 2 · p. 8, §4.2 · p. 8
- Two sentence ratings were excluded because of textual overlap with documents, and one fluency rater’s document ratings were replaced after poor spam-item performance; results were analysed with two-tailed Sign Tests.The overlap meant sentence-level decisions could have been informed by access to the full documents.Source: §4.2 · p. 8
- For adequacy, raters showed no significant preference between MT1 and HA on isolated sentences, but significantly preferred HA when assessing whole documents.Document-level context cues allow raters to get a signal on adequacy.Source: §4.2 · p. 8
4.3 Discussion
- For adequacy, whole-document assessment yielded a significant preference for human translation, unlike randomly ordered single-sentence assessment.The authors emphasize linguistic context’s importance in human evaluation of MT.Source: §4.3 · p. 9
- Document-level review exposes errors that are hard or impossible to spot sentence by sentence, including inconsistent rendering of a cultural festival by MT1 where HA uses one translation.The example compares two consecutive sentences from a Chinese news article.Source: §4.3 · p. 9
5. Reference Translations
This section tests how reference-translation quality and directionality affect judgments of human–machine parity. It finds that aggressive editing can lower adequacy, while translated-English source text gives MT higher human-evaluation scores than original Chinese.
5.1 Quality
- The quality analyses used four professional translators per condition and added a blinded, document-context audit of 150 sampled sentences with a Context error category.The audit compared MT1, HA, and HB and marked errors apparent only in larger context.Source: §5.1 · pp. 10–11
- HA was judged significantly more adequate than HB, while raters showed no significant fluency difference between them.Aggressively edited HB was less adequate; raters did not prefer human over machine translation on adequacy, but preferred human translation on fluency.Source: §5.1 · p. 10, Table 4 · p. 11
- MT1 had significantly more Incorrect Word, Named Entity, and Word Order errors, while HB had more Missing Word (Semantics) errors than HA and MT1.The audit covered 150 sampled sentences and classified errors by type.Source: §5.1 · pp. 11, 14, Table 5 · p. 12
- MT1 showed more Collocation and Context errors numerically, but the differences were not significant, likely because there were few examples.These categories indicate potential difficulty translating words dependent on longer-range context.Source: §5.1 · p. 14
- The error audit agreed with the pairwise rankings: HA was preferred over MT1 and had fewer errors, while HB had more deletions but resembled HA on most other categories.HB performed better than MT1 in most other error categories.Source: §5.1 · p. 14
5.2 Directionality
- MT systems score higher in human evaluation on translationese sources than on original sources, and translationese has lower lexical variety.For the compared subsets, Chinese-source TTR was 13% higher than English-source TTR.Source: Table 7 · p. 15, §5.2 · pp. 15–16
6. Recommendations
The experiments show that current best-practice evaluations can miss errors in strong MT, motivating changes to how human–machine parity is assessed. The recommendations address raters, evaluation context, fluency, reference editing, and source-text origin.
6. Recommendations
- Current best-practice evaluations can miss errors in strong MT, and the experiments show that MT has not reached professional human translation quality.Source: §6 · p. 16
- Professional translators as raters identify differences between human and machine translation that non-expert raters may neglect.Source: §6 · p. 16
- Evaluating full documents rather than isolated sentences exposes coherence and cohesion errors and increases the rating gap between human and machine translation.Source: §6 · p. 16
- Assess fluency as well as adequacy: raters prefer human translation for fluency, while sentence-level adequacy judgments show no significant human–machine difference.Source: §6 · p. 16
- Aggressive fluency editing can make human reference translations less accurate, to the point that their adequacy becomes indistinguishable from MT.Source: §6 · pp. 16–17
- Use original source texts: raters significantly preferred human translations for original texts, but not for texts translated from another language.Source: §6 · p. 17
- The recommendations build on prior work, and WMT 2019 adopted original source texts and document-level context in its evaluation protocol.Source: §6 · p. 17
- The evidence is limited mainly to a small number of raters, one news domain, and Chinese-to-English translation, so broader testing is needed.Source: §6 · p. 17
7. Conclusion
The paper concludes that reported human–machine parity reflected weaknesses in evaluation design: professional translations contained fewer errors, and evaluation outcomes varied with raters, context, and reference creation. It recommends revisiting human MT assessment while noting its cost and the need for lower-cost valid protocols.
7. Conclusion
- Professional human translations had significantly fewer incorrect words, omissions, mistranslated names, and word-order errors than the strong MT output.Source: §7 · p. 17
- Recent parity findings are attributed to weaknesses in human evaluation design, with outcomes affected by raters, linguistic context, and reference translations.Source: §7 · pp. 17–18
- 11–22% for segment-level evaluation, 24% for segment-level evaluation with document-level context, and 39% for document-level evaluation were the absolute differences in average standardized human scores between HUMAN and CUNI-Transformer-T2T-2018.Source: §7 · p. 17
- Human translation was significantly more accurate in full-document evaluation, while fluency judgments favored humans for both documents and isolated sentences.Source: §7 · p. 18
- The authors recommend revisiting human MT assessment, while noting the high cost of professional raters and full-document evaluation and calling for valid lower-cost alternatives.Source: §7 · p. 18