Source-linked AI summary
Achieving Human Parity on Automatic Chinese to English News Translation
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, Ming Zhou
TL;DR
The paper asks how machine translation can be rigorously defined and measured against human quality, given limitations of reference-based evaluation. It presents Microsoft’s Chinese-to-English news translation system and its training and decoding approaches, finding parity with professional human translations and higher quality than crowd-sourced references.
Problem
Human parity is difficult to define and measure because valid translations differ and reference-based metrics can inherit poor-reference bias.
Method
The paper defines parity statistically and combines dual learning, joint training, two-pass deliberation, agreement regularization, and system combination in NMT.
Results
The system reaches parity with professional human translations on WMT 2017 Chinese-to-English news translation and exceeds crowd-sourced references.
Takeaways & Limitations
Human parity is assessed as statistical indistinguishability from human translations rather than superiority or error-free performance.
Takeaways & Limitations
The reported parity is tied to a specific test set and does not automatically generalize to other domains or language pairs, especially with limited data and resources.
Abstract
from arXiv · showhide
Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question naturally arises whether such systems can approach or achieve parity with human translations. In this paper, we first address the problem of how to define and accurately measure human parity in translation. We then describe Microsoft's machine translation system and measure the quality of its translations on the widely used WMT 2017 news translation task from Chinese to English. We find that our latest neural machine translation system has reached a new state-of-the-art, and that the translation quality is at human parity when compared to professional human translations. We also find that it significantly exceeds the quality of crowd-sourced non-professional translations.
1 Introduction
Machine translation has improved substantially, but quality still varies across language pairs, domains, and genres with training-data availability. This paper targets human parity for Chinese-to-English news translation while cautioning against automatic generalization.
- Translation quality varies across language pairs, domains, and genres, broadly tracking the availability of training data.
- Reference-based metrics struggle to assess human or near-human translation because equally correct translations can differ substantially.
- Crowd-sourced references can be poor because workers may use online machine translation or lack sufficient command of a language.
- The paper describes methods for defining and measuring human quality, its system architecture, data and experiments, and evaluation results.
2 Human Parity on Translation
The paper defines human parity statistically as no significant quality difference between machine and corresponding human translations. It uses direct human assessment to evaluate parity while preserving important scope limits.
- Defining Human Parity: Human parity means bilingual judges consider machine and human candidate translations equivalent.
- Defining Human Parity: Human parity is reached when machine and human translation scores show no statistically significant difference on a test set.
- Scope and Caveats: Parity does not mean outperforming humans or producing error-free translations, and parity on one test set does not automatically generalize.
- Human Evaluation: The evaluation uses direct assessment and degraded outputs to reduce reference bias and identify unreliable crowd workers.
- Human Evaluation: Source-based direct assessment asks bilingual annotators to rate how accurately a candidate conveys source semantics on a 0–100 slider.
- Human Evaluation: Human scores are standardized as z-scores, averaged at segment and system levels, and clustered using statistical significance testing.
3.1 Neural Machine Translation
Neural machine translation generally uses attention-based encoder-decoder models to represent inputs and generate outputs. Convolutional and Transformer architectures reduce recurrent processing while modeling dependencies differently.
- Encoder-Decoder NMT: Attentional NMT models p(y|x) with an encoder that represents the input and a decoder that generates the translated sequence.
- Attention: Attention computes context from encoder outputs using the decoder’s internal state and preceding output information.
- Architectures: ConvS2S and Transformer architectures avoid recurrent decoder states, improving parallelizability and reducing sensitivity to sequence length.
- Architectures: ConvS2S models nearby dependencies in lower convolutional layers and longer-range dependencies in upper layers, with attention at each decoder layer.
- Architectures: Transformers replace convolutions with multi-head self-attention, modeling dependencies across source positions without recurrent processing.
3.2 Reaching Human Parity
The paper addresses limitations of autoregressive NMT through dual learning, joint use of bilingual and monolingual data, and methods that incorporate right-context information. Complementary systems are combined to attain human parity.
- Overview: The paper targets human parity on large-scale Chinese-to-English news data while addressing limitations of the prevailing NMT paradigm.
- Dual Learning: Dual learning exploits translation duality so models learn from both source-to-target and target-to-source directions using supervised and unsupervised data.
- Decoding and Regularization: Deliberation Networks refine translations with two-pass decoding, while agreement regularization encourages consensus between left-to-right and right-to-left decoding.
- Data Quality: Data selection and filtering address NMT vulnerability to noisy training data, rare occurrences, and general training-data quality.
- System Combination: System combination exploits complementarity among systems and ultimately supports human parity.
3.3 Exploiting the Dual Nature of Translation
The paper exploits translation duality by jointly training source-to-target and target-to-source models, using each direction to improve the other with monolingual and bilingual data. Dual unsupervised learning supplies reconstruction-based supervision from monolingual data, while dual supervised learning regularizes bilingual-model consistency.
- Dual learning trains Chinese-to-English and English-to-Chinese models as complementary primal and dual translation systems.
- Dual unsupervised learning uses sampled translations and reconstruction likelihoods from the dual model to supervise translation with monolingual data.Monte Carlo sampling approximates the expectation over possible translations, and the dual model can be updated symmetrically using monolingual data in the other language.
- Dual supervised learning adds a loss that encourages joint-probability consistency between the two models on bilingual sentence pairs.The regularizer targets consistency between p(x)p(y|x) and p(y)p(x|y), using empirical marginal distributions from language models.
- The combined training framework applies dual unsupervised and dual supervised learning to both monolingual and bilingual corpora.
- Joint training generates weighted pseudo-parallel data in both directions and repeatedly updates the models using original bilingual data and synthetic pairs.The two conditional models act as each other’s pseudo-training-data generators; n-best translations are probability-weighted to reduce noise, and synthetic data are regenerated iteratively.
3.4 Beyond the Left-to-Right Bias
The paper addresses exposure bias by giving translation access to broader target-side context and by encouraging agreement between left-to-right and right-to-left models. Its deliberation architecture performs two-pass refinement, while agreement regularization uses the opposite decoding direction as a training signal.
- 3.4 Beyond the Left-to-Right Bias: Exposure bias arises because autoregressive decoding can amplify previous errors and mislead subsequent generation.The paper proposes two remedies: two-pass deliberation decoding and agreement regularization between left-to-right and right-to-left models.
- 3.4.1 Deliberation Networks: Deliberation Networks generate an initial translation and then refine it using the first-pass output to provide global target-side information.The second-pass decoder receives the first-pass translation, enabling access to right context during sentence generation.
- 3.4.1 Deliberation Networks: The deliberation network contains an encoder, a first-pass decoder, and a second-pass decoder connected through attention-based information flow.The encoder and first-pass decoder use stacked self-attention layers, while policy-gradient algorithms jointly optimize the three components.
- 3.4.2 Agreement Regularization: Agreement regularization uses right-to-left model scores to identify problematic left-to-right continuations and provides a reciprocal signal between decoding directions.The training objective introduces two KL-divergence regularization terms, and the models iteratively improve one another through pseudo-corpora and weighted samples.
- 3.4.2 Agreement Regularization: The unified system trains four source-to-target and target-to-source models whose weighted pseudo-corpora and agreement regularization mutually improve the models until convergence.
3.5 Data Selection and Filtering
The system selects relevant, cleaner parallel data for NMT using bilingual sentence representations learned from a bidirectional Chinese-English model. Filtering low-similarity sentence pairs reduces training data while improving accuracy.
- 3.5 Data Selection and Filtering: The proposed data-selection method addresses relevance and noise jointly, unlike standard SMT selection methods that reduce NMT training data without improving quality.Its bilingual sentence representation supports both filtering noisy data and selecting task-relevant data.
- 3.5 Data Selection and Filtering: A unified bilingual NMT system translates in both directions to learn representations for Chinese and English sentences.The model is trained on a high-quality, relevant subset so cleaner and more relevant sentences receive better representations.
- 3.5 Data Selection and Filtering: Each sentence vector is computed as the mean of its word-level contextual vectors from the encoder.The model omits language markers to encourage similar representations for Chinese and English.
- 3.5 Data Selection and Filtering: Low-similarity sentence pairs are rejected using cross-lingual sentence similarity, drastically reducing training data while significantly improving accuracy.The same procedure can filter noisy data and select relevant data.
3.6 System Combination and Re-ranking
The system combines n-best hypotheses from multiple translation systems and trains a k-best MIRA re-ranker on the validation set. Re-ranking uses system, language-model, bidirectional, and sentence-similarity features.
- 3.6 System Combination and Re-ranking: N-best hypotheses from all systems are combined and re-ranked using k-best MIRA trained on the validation set.K-best MIRA is a batch-tuned version of the margin-based classification algorithm MIRA.
- 3.6 System Combination and Re-ranking: The re-ranker includes original system scores, an English 5-gram language-model score, and reverse-direction system scores.The language model is trained on English news crawled in 2015 and 2016.
- 3.6 System Combination and Re-ranking: Additional features measure cross-lingual similarity between the source and each hypothesis, between hypotheses, and after round-trip translation.These include STSV, R2LSV, and E2ZSV features.
4 Experiments
The experiments evaluate Chinese-to-English systems on WMT17 using parallel and monolingual data, then test data enhancement, data selection, and system combination. These approaches improve BLEU, with SentVec filtering and heterogeneous system combination producing strong results.
- Data and setup: 18M bilingual sentence pairs remained after filtering, alongside about 7M Chinese and English monolingual sentences retained for dual learning and back-translation.The initial data included News Commentary, UN Parallel, and CWMT corpora; monolingual data came from News Crawl and Common Crawl.
- Data and setup: The evaluation uses Newsdev2017 for development and Newstest2017 for testing, with Jieba, Moses tokenization, and 32K BPE merge operations.Source and target vocabularies contain 44K and 33K sub-word tokens, respectively.
- Model experiments: 27.40 BLEU was achieved by DLDN, improving 0.89 BLEU over Dual Learning and surpassing the WMT17 challenge’s best ensemble result of 26.40.Dual Learning itself achieved 26.51 BLEU, a 0.94-point improvement over BT.
- Data enhancement: Agreement regularization improved over BT by 1.34 BLEU points, while adding joint training increased the gain to 1.81 BLEU points.The unified ARJT framework integrates agreement regularization, back translation, and joint training.
- Data selection: SentVec similarity filtering improved systems by up to 1.5 BLEU points over Base8K and nearly 1 BLEU point over equally sized CED-selected data.The authors attribute this to filtering noisy or low-quality translations and selecting data relevant to CWMT.
- System combination: Combining heterogeneous systems achieved the highest results, with complementary systems benefiting from n-best-list combination and feature-based reranking.The most helpful scoring features included SY SScore, LMScore, R2Lscore, R2LSV, and E2ZSV.
5 Human Evaluation Results
The evaluation uses source-based direct assessment with full system coverage and three annotators per task, finding Microsoft research systems statistically comparable to professional human translations across multiple test subsets.
- Evaluation methodology: Source-based direct assessment evaluates candidate translations against the source text rather than reference translations.The campaign used bilingual crowd workers and an updated version of Appraise.
- Evaluation methodology: Three annotators scored every segment for every system, adding redundancy and fully comparable scores for external validation.Full system coverage ensured that every segment received human scores for all systems under investigation.
- Subset-1 results: All three research systems achieved human parity with Reference-HT in the first Subset-1 evaluation round, with at least n ≥609 assessments per system.They also outperformed Sogou and both Reference-WMT and Reference-PE in that round.
- Subset-1 results: Across the second and third Subset-1 rounds, Reference-HT, Reference-PE, and Combo-4 through Combo-6 formed the top cluster, outperforming Sogou and Reference-WMT.Online-A-1710 and Online-B-1710 performed worst in both rounds.
- Subset-2 results: On Subset-2, Reference-HT and the three research systems were not significantly different, while all four outperformed Reference-PE.At least n ≥607 assessments were collected per system.
- Subsets 3–4 results: On Subsets 3 and 4, the research systems remained in the top cluster with human references, although Reference-PE ranked highest by z score on Subset-3.Subset-4 found no significant quality difference among Reference-HT, Reference-PE, and Combo-4 through Combo-6.
- Evaluation resources: The released evaluation data supports external validation of the human-parity claim and provides two additional human references.The benchmark reference was included because it was known to contain errors.
6 Human Analysis
A preliminary error analysis of the best-performing Combo-6 system identifies four dominant error categories, each affecting roughly 5% of sampled sentences.
- Error analysis: The analysis randomly sampled 500 Combo-6 outputs and annotated nine predefined translation error categories.The categories included missing words, repetition, named entities, word order, incorrect words, unknown words, collocations, factoids, and ungrammatical output.
- Error distribution: Missing words, incorrect words, ungrammatical output, and named entities were the four major error types.The named-entity category was further subdivided into person, location, and organization errors.
- Error distribution: Each of the four major error types appeared in roughly 5% of sampled sentences.The paper identifies these categories as opportunities for further machine-translation improvement.
7 Discussion and Future Work
The paper reports human parity for Microsoft’s Chinese-to-English news translation system while emphasizing direct human evaluation and continued limits outside data-rich settings.
- Discussion: The latest Microsoft system reached a new state-of-the-art and parity with professional human translations on WMT 2017 Chinese-to-English news translation.It also exceeded the quality of crowd-sourced references.
- System techniques: The system exploited parallel and monolingual data through dual training, while addressing exposure bias with deliberation decoding, agreement regularization, and joint training.These approaches jointly trained source-to-target, target-to-source, left-to-right, and right-to-left systems.
- Evaluation: Direct human annotation was used because automatic reference-based metrics become increasingly problematic as translation quality improves.The evaluation measured both human and machine translations.
- Future work: Further progress remains necessary, especially for domains and language pairs without huge amounts of available data.The paper frames its achievement specifically around WMT2017 Chinese-to-English news translation.