Source-linked AI summary
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation
Antonio Toral, Sheila Castilho, Ke Hu, Andy Way
TL;DR
The paper revisits Hassan et al.’s claim that Chinese-to-English NMT reached human parity by testing original-language sources, evaluator expertise, and inter-sentential context. It finds that parity is not supported for original Chinese text, while expert judgments better distinguish human from machine translations, and it identifies translation problems and evaluation-scope limits.
Problem
A prior study claimed human parity for Chinese-to-English news translation, but its evaluation did not account for source-text origin, evaluator translation proficiency, or inter-sentential context.
Method
The paper reassesses the claim using a modified human evaluation that varies source-text origin, evaluator proficiency, and access to inter-sentential context.
Results
Human parity has not been achieved for the subset originally written in Chinese; professional translators show higher agreement and wider human–machine discrimination than non-experts.
Takeaways & Limitations
Human evaluation of MT should account for original source language, evaluator expertise, and inter-sentential context, while scrutinizing human reference translations.
Takeaways & Limitations
The conclusions and recommendations derive from experiments on one language direction and five evaluators.
Abstract
from arXiv · showhide
We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that human parity has not been achieved. We compare the judgments of professional translators against those of non-experts and discover that those of the experts result in higher inter-annotator agreement and better discrimination between human and machine translations. In addition, we analyse the human translations of the test set and identify important translation issues. Finally, based on these findings, we provide a set of recommendations for future human evaluations of MT.
1 Introduction
The paper reassesses claims that Chinese-to-English NMT achieved human parity by examining weaknesses in the prior evaluation and conducting a modified evaluation. It also investigates translation quality and proposes recommendations for future MT evaluations.
- Hassan et al. claimed that Microsoft achieved human parity for Chinese-to-English news translation, amid broader claims of near-human NMT quality.
- The authors use Hassan et al.’s released experimental data to reassess Microsoft’s human-parity claim under what they consider a fairer evaluation setting.
- The paper examines the language of the original source text, evaluator translation competence, and available linguistic context as potential evaluation issues.
- They analyze problems in the test set’s human translations with help from English- and Chinese-native speakers.
- The paper concludes with recommendations for future human evaluations and warnings about overhyping MT capabilities.
2 Potential Issues
The paper identifies three evaluation issues: mixed original-language sources, uncertain evaluator expertise, and sentence-level assessment without inter-sentential context. These factors can affect judgments of MT quality and human parity.
- 2.1 Original Language of the Source Text: 2,001 sentence pairs comprise newstest2017, with half originally written in Chinese and half originally written in English.WMT organisers manually translated each subset into the other language to create the parallel test set.
- 2.1 Original Language of the Source Text: Translationese may make English-origin sentences easier to translate than Chinese-origin sentences because translated text tends to be simpler, more explicit, and more normalized.
- 2.2 Human Evaluators: Bilingual crowd-worker evaluations provided no evaluator-selection details or inter-annotator agreement figures, leaving translation expertise insufficiently characterized.The paper notes that conventional Kappa is unsuitable for their continuous evaluation scale.
- 2.2 Human Evaluators: Prior evidence suggests crowd workers may accept MT output more readily than professional translators, assigning higher fluency and adequacy scores and performing little post-editing.
- 2.2 Human Evaluators: The authors therefore compare professional translators with bilingual speakers lacking formal translation qualifications.
- 2.3 Context: Randomized sentence-level evaluation omits document context, although correct translation of news stories requires inter-sentential relations that the MT system does not model.
- 2.3 Context: Document-level evaluation in related work produced a stronger preference for human translation than isolated-sentence evaluation.
3 Evaluation
The evaluation reassesses Chinese-to-English MT using pairwise rankings that control for original source language, evaluator proficiency, and document context. It finds that translationese affects system comparisons, while experts provide stronger discrimination and agreement than non-experts.
- Experimental Setup: The study compares Microsoft human translation (HT), Microsoft MT (MS), and Google Translate (GG) using pairwise rankings of source sentences and their three English translations.Evaluators ranked consecutive sentences from randomized documents, rather than judging isolated sentences in random order.
- Experimental Setup: The evaluation covered 503 sentences from 49 documents and collected 6,675 pairwise judgments from two professional translators and three bilingual non-experts.All evaluators completed nearly all assigned material, with limited omissions by one non-expert and one professional translator.
- Original Language: When sentences were grouped by original language, HT significantly outperformed MS for originally Chinese sentences, but not for sentences originally written in English.The result supports the claim that translationese in the source can distort evaluation in MS’s favour.
- Original Language: In Hassan et al.’s evaluation, HT exceeded MS by around 0.05 absolute points for originally Chinese sentences, whereas the gap disappeared for originally English sentences.Compared with ZH, MT scores increased substantially on EN translationese: GG by over 10% absolute and MS by over 6% absolute.
- Evaluators: Expert evaluations separated HT and MS more strongly than non-expert evaluations, with score gaps of 2.2 versus 1.2 for experts and 1.3 versus 0.9 for non-experts.The authors associate the smaller non-expert gaps with overlooking translation subtleties and errors.
- Evaluators: Professional translators reached higher inter-annotator agreement than non-experts: κ = 0.254 versus κ = 0.130, a 95% relative difference.All three pairwise non-expert coefficients were below the professional-translator agreement coefficient.
4 Analyses
The analysis identifies substantial quality problems in the test set’s human translations, including grammatical errors, mistranslations, disfluency, and possible inexperienced or poorly post-edited translation. These issues affected how professional translators and non-experts judged the human and machine outputs.
- English Original: The English originals were generally fluent and grammatical, apart from a few errors that did not affect their Chinese translations or subsequent backtranslations.The errors included “on the air” instead of “on air” and a missing “that” after “apologize.”
- Chinese Translation: Most WMT Chinese translations contained grammatical errors, mistranslated proper nouns, or disfluency despite often transferring the source meaning accurately.Word-order problems could make a translation disfluent even when its meaning was fully preserved.
- Chinese Translation: The authors speculate that the Chinese translations may have been produced by inexperienced translators or through poorly post-edited machine translation.They argue that the observed errors are inconsistent with the expected care of experienced professional translators.
- English Translation: Many Microsoft English references contained grammatical errors and mistranslations, including incorrect prepositions, omitted information, and altered meanings.One reference omitted “loft” and described an architectural design makeover rather than an interior makeover.
- Evaluator Judgments: Professional translators judged several flawed human references worse than Microsoft output, whereas non-experts were more likely to rate them as good as or better than machine translations.The examples provide qualitative evidence that non-experts may be more tolerant of translation errors than professionals.
- Dataset Quality: The test-set translations may themselves have been inexperienced human translations or poorly post-edited machine translations, and translators lacked access to the original English for one direction.For the Chinese-to-English references, the Chinese translation was used as the source, which could constrain the resulting English translation.
5 Conclusions and Future Work
Reassessing Chinese-to-English human parity while accounting for source-language origin, evaluator proficiency, and intersentential context shows that the original parity claim does not hold for originally written Chinese text. The paper also identifies translation-quality problems and recommends stricter evaluation practices.
- Findings: For source text originally written in Chinese, human parity was not achieved: human and machine translations differed significantly in both evaluations.
- Findings: Translationese input produced higher MT scores than original text in Microsoft’s human evaluation.
- Findings: Professional translators produced higher inter-annotator agreement and a wider human–machine translation gap than non-experts.
- Findings: Problems in WMT and Microsoft human translations suggest non-expert translation and possible post-editing of MT output.
- Recommendations: The paper recommends matching the test set’s source language to the language in which its source text was originally written.
- Recommendations: Human evaluation should use professional translators and evaluate whole documents rather than isolated sentences.
- Recommendations: Test sets should be translated from scratch by experienced professional translators.
- Implications: The authors caution that faulty test sets can make higher automatic scores indicate closeness to flawed references rather than better translation quality.
Appendix: Evaluator Instructions
The appendix instructs evaluators to rank three translations from best to worst for a source segment, allowing ties and using the highest possible rank. Each task corresponds to one document of up to 50 sentences.
- Ranking task: Evaluators rank three translations from best to worst given a source segment.
- Ranking task: A translation receives a higher rank than another when it is judged better.
- Ties: Translations judged equal in quality receive the same rank.
- Ties: When tied translations outrank another translation, evaluators use the highest possible ranks rather than lower sequential rankings.
- Document-level evaluation: Each task covers one document containing up to 50 sentences, preferably annotated in one sitting.