Source-linked AI summary
A Challenge Set Approach to Evaluating Machine Translation
Pierre Isabelle, Colin Cherry, George Foster
TL;DR
Neural machine translation improved quality, but standard metrics and opaque models make its remaining linguistic weaknesses difficult to identify. The paper uses manually designed English–French challenge sets to test specific structural divergences in PBMT and NMT outputs. Neural systems outperform phrase-based systems across clear differences, yet still struggle with idioms and incomplete generalizations.
Problem
Improving neural MT makes surface-based metrics less informative and leaves its specific linguistic strengths and weaknesses difficult to diagnose.
Method
The paper manually constructs a small English–French challenge set whose sentences each probe one structural divergence, then evaluates phrase-based and neural outputs.
Results
68% success rate made GNMT best overall on the challenge set, while both NMT systems clearly outperformed PBMT across the three divergence categories.
Takeaways & Limitations
Challenge sets supplement traditional evaluation by providing fine-grained evidence about which linguistic phenomena MT systems handle successfully or poorly.
Takeaways & Limitations
The challenge set excludes degradation on long sentences and reveals that neural generalizations can remain incomplete or brittle.
Abstract
from arXiv · showhide
Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system's capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.
1 Introduction
Neural machine translation improves quality but makes fine-grained diagnosis harder and exposes weaknesses in surface-based metrics. The paper proposes challenge sets to probe specific linguistic divergences and analyzes English–French phrase-based and neural systems.
- Neural systems approach human performance more closely than previous statistical systems on easy language pairs such as English–French.
- BLEU relies on surface matching and may become less effective as neural translation quality improves.
- Challenge sets use expert-designed difficult sentences to probe an MT system’s ability to handle particular linguistic phenomena.
- The English–French set was constructed around linguistic divergences, especially those exposed by French morphology, and evaluated in PBMT, NMT, and GNMT systems.
- The study contributes an annotated, machine-readable challenge set, evidence that NMT can outperform PBMT despite small BLEU differences, and analysis of remaining NMT challenges.
2 Related Work
Prior work used broad metrics, corpus-based error analyses, and contrastive pairs to evaluate NMT. This paper instead manually evaluates a smaller set of hand-constructed sentences targeting specific linguistic divergences.
- Broad evaluations reported strong NMT results across translation directions using BLEU and human judgments.
- Corpus-based analyses found NMT advantages in morphology and word order but greater degradation as sentence length increased.
- Contrastive evaluation tested whether NMT probabilities preferred original references over versions containing targeted errors.
- The proposed approach uses manually constructed sentences designed as canonical, difficult examples of specific linguistic phenomena rather than sentences drawn from existing bitext.
- Because the set is small, the study manually assesses actual system outputs, avoiding a limitation of rankings based only on contrastive pairs.
3 Challenge Set Evaluation
The challenge set targets structural divergences in English–French translation with short, focused sentences and direct manual judgments. It covers morpho-syntactic, lexico-syntactic, and syntactic phenomena, while remaining complementary to broad evaluation.
- Challenge set design: The English–French set contains 108 sentences spanning morpho-syntactic, lexico-syntactic, and purely syntactic divergences.
- Morpho-syntactic divergences: The set tests whether systems recover target-language morphology when the source under-specifies information available from context, including agreement and subjunctive marking.
- Lexico-syntactic divergences: Lexico-syntactic cases include argument switching, where a correct translation requires reversing source arguments, and crossing-motion verb alternations.
- Syntactic divergences: Syntactic cases include French clitic pronoun placement, obligatory fronting of prepositions under WH-movement, and French middle-voice constructions.
- Evaluation methodology: Each sentence targets one divergence subtype, uses a highlighted reference translation, and asks whether the system bridged that divergence correctly.
- Evaluation methodology: The approach complements overall translation evaluation, but may count translations with errors outside the targeted divergence as successes.
4 Machine Translation Systems
The study compares state-of-the-art English–French PBMT and NMT systems trained on WMT 2014 data, alongside Google’s production GNMT. The configurations differ in language-model use, architecture, scale, and training resources.
- Data: The experiments train state-of-the-art neural and phrase-based English–French systems on WMT 2014 data.
- Phrase-based systems: The PBMT baseline combines phrase probabilities, lexical probabilities, reordering features, an NNJM model, sparse features, penalties, and French 5-gram language models.
- Phrase-based systems: PBMT-1 matches the NMT data conditions, whereas PBMT-2 additionally uses a French monolingual language model.
- Neural systems: The in-house NMT system uses attention with a single-layer sequence-to-sequence architecture, 512-dimensional embeddings, 1024-dimensional states, and 172M parameters.
- Neural systems: GNMT was evaluated as a production system with at least eight encoder and eight decoder layers and training corpora two to three orders of magnitude larger than WMT.
5 Experiments
On the challenge set, neural MT systems substantially outperform PBMT across morpho-syntactic, lexico-syntactic, and purely syntactic divergences, while still failing on several difficult constructions. Performance also exposes limitations of BLEU and shows that data scale matters, especially for Google NMT.
- Quantitative comparison: The two NMT systems outperform both PBMT systems in all three divergence categories, with GNMT best overall at 68% success.PBMT performs especially poorly on morpho-syntactic and purely syntactic divergences.
- Quantitative comparison: A 0.4 BLEU-point gap between PBMT-2 and NMT corresponds to a 21% challenge-set gain, whereas a 2.3-point PBMT gap corresponds to only 1%.These results indicate poor correlation between WMT BLEU and challenge-set performance.
- Strengths of neural MT: Neural systems improve morpho-syntactic performance from 16% to 72% over PBMT-1, particularly on complex subject-verb agreement.They correctly handle agreement across intervening noun phrases, coordinated verb phrases, and coordinated subjects.
- Strengths of neural MT: NMT better handles lexico-syntactic divergences, including double-object verbs, overlapping subcategorization frames, and English infinitival complements rendered as French finite clauses.The systems perform the required adjustments for double-object verbs and more often distinguish noun-phrase from sentential complements.
- Strengths of neural MT: NMT also better handles purely syntactic divergences, including transforming English pronouns into correctly positioned and case-inflected French proclitics.GNMT additionally succeeds on tag questions, stranded prepositions, many inalienable-possession cases, and zero relative pronouns.
- Strengths of neural MT: The large gap between in-house and Google NMT indicates that neural systems are extremely data hungry, although sufficient data can enable difficult phenomena such as some stranded-preposition cases.The challenge set also reveals that NMT lacks raw memorization capabilities on some examples.
- Weaknesses of neural MT: Despite its advantage over PBMT, GNMT remains below 70% on the challenge set and all systems fail several constructions, including subject control, crossing movement verbs, and middle voice.NMT also misses some generalizations, performs disappointingly on noun compounds, and shows incomplete agreement and subjunctive-trigger handling.
6 Conclusions
The challenge-set evaluation reveals both broad neural MT gains over PBMT and persistent weaknesses, especially idioms and brittle generalization.
- Challenge sets stress-test MT systems on difficult linguistic material while classifying their successes and failures finely.
- Except for idiom processing, every clear difference between PBMT and neural MT favored neural systems.
- Neural MT appears to overcome n-gram language-model limitations in agreement, double-object verbs, subcategorization frames, and stranded-preposition WH-movement.
- Current neural MT still struggles with idiomatic phrases, even with massive amounts of training data.
- Neural generalizations can remain brittle: systems may master agreement or inalienable possession yet fail on obvious instances.
7 Future Work
Future work targets scalability, portability, and training improvements for challenge-set evaluation, while preserving its fine-grained human assessment and released resources.
- Challenge-set insights are intended to inspire MT research and highlight unresolved linguistic issues even for English–French translation.
- Human judgments precisely assess targeted phenomena but do not scale to large sets and require native speakers for replication.
- Automatic judgments could be meta-evaluated against the challenge set’s existing human judgments as a gold standard.
- Automatically constructing challenge sets for new language pairs would reduce the need for in-depth expertise about structural divergences.
- Future training directions include curricula demonstrating divergences or network changes designed to capture them.
- The released machine-readable judgments support future uses of the challenge-set data.
S-V agreement, feature calculus on coordinated source
The challenge set probes French agreement, participle morphology, mood, and argument structure, exposing strong performance on some patterns alongside systematic failures.
- S-V agreement, feature calculus on coordinated source: NMT and Google correctly handle several coordinated-subject agreement patterns, while PBMT-1 often produces incorrect number or gender marking.
- S-V agreement, feature calculus on coordinated source: For coordinated subjects involving grammatical person, NMT succeeds on two examples, while PBMT-1 and Google fail across the tested cases.
- S-V agreement, feature calculus on coordinated source: Past-participle agreement after avoir depends on whether a preceding object controls agreement; Google succeeds on two of three examples, whereas NMT succeeds on two.
- S-V agreement, feature calculus on coordinated source: Only Google correctly handles the subjunctive trigger in “malheureux que”; all systems miss “à condition que,” while all succeed on “demander que.”
- S-V agreement, feature calculus on coordinated source: All systems fail the argument-switch examples, misrepresenting the experiencer and object in the French rendering of “miss.”
Double-object verbs
The challenge set tests whether systems preserve meanings across English constructions that require French restructuring, revealing selective neural and Google strengths.
- Double-object verbs: Double-object constructions require French renderings that correctly preserve gift and recipient roles.
- Double-object verbs: NMT and Google correctly translate two of three double-object examples, while PBMT-1 correctly handles only the story example.
- Double-object verbs: The English meaning of “fail to” is correctly rendered by NMT in one example and by all systems in two others.
- Double-object verbs: None of the systems correctly restructure the tested manner-of-movement verbs with path arguments or the nonce verb “guitared.”
- Double-object verbs: Systems generally choose between “savoir” and “connaître” correctly, except PBMT-1 on clauses such as “knows this story is hard to believe.”
- Double-object verbs: Finite-clause restructuring for English NP-to-VP complements is mixed: Google succeeds on two examples, NMT on two, and PBMT-1 on one.
Noun Compounds
The noun-compound challenge tests whether English compounds receive the correct French preposition. Across examples, PBMT and NMT each handle some compounds but make distinct prepositional and ordering errors.
- Noun Compounds: The challenge asks whether English nominal compounds receive the correct preposition in French.
- Noun Compounds: NMT correctly translated meat, steak, water, and cloth filters, but mistranslated butter, juice, and tea filters.
- Noun Compounds: PBMT correctly translated butter, water, tea, cloth, and metal filters, but mistranslated meat, steak, and juice filters.
- Noun Compounds: Both NMT and PBMT correctly translated the cloth and water filter compounds, while both failed on different individual compounds.
Common idioms
The challenge set tests whether English idioms and related question forms are rendered as natural French expressions. PBMT and NMT often preserve literal wording or distort syntax, while Google succeeds on several idioms and questions.
- Common idioms: The idiom challenge tests whether English expressions are rendered with suitable French idiomatic expressions.
- Common idioms: Google correctly translated “Stop beating around the bush,” while PBMT and NMT produced non-equivalent literal or distorted expressions.
- Common idioms: PBMT correctly translated the cart-before-horse and nail-on-the-head idioms, whereas NMT and Google failed on both examples.
- Common idioms: Google correctly translated one version of “crying over spilt milk,” but all three systems failed on the alternate version.
- Common idioms: NMT correctly rendered the supplied English questions, although it used “Est-ce que” for two examples instead of the reference inversion.
- Common idioms: Google correctly rendered all three question examples, while PBMT failed on the first two and succeeded on the third.
WH-MVT and stranded preps
These challenges probe French restructuring of stranded prepositions, adverb-triggered inversion, generic statements, conditionals, pronouns, and ordinal–numeral order. The examples show frequent errors across PBMT, NMT, and Google, with some localized successes.
- WH-MVT and stranded preps: The stranded-preposition challenge tests whether French moves English sentence-final prepositions into appropriate constructions.
- WH-MVT and stranded preps: Google correctly translated all six stranded-preposition examples, while PBMT and NMT failed on every listed example.
- WH-MVT and stranded preps: The inversion challenge tests whether adverb-triggered subject–verb inversion is rendered in French.
- WH-MVT and stranded preps: Google succeeded on the “Never before” example but failed on the “Rarely” and “Nowhere” examples; NMT failed on all three.
- WH-MVT and stranded preps: Generic-statement examples were difficult: PBMT, NMT, and Google each failed on all three supplied translations.
- WH-MVT and stranded preps: For fronted “should,” French normally uses “si” with imperfect tense; Google succeeded only on the first of three examples.
- WH-MVT and stranded preps: All three systems correctly ordered the ordinals and numerals in the first two examples, while each produced a marked error in the third.
Inalienable possession
The section tests French choices for body-part determiners and reflexive pronouns, alongside zero relative pronouns. Systems show recurring difficulty with possession and relative-clause restructuring.
- Inalienable possession: The inalienable-possession challenge tests both body-part determiners and whether a reflexive pronoun precedes the verb.
- Inalienable possession: PBMT and Google correctly handled “He washed his hands,” while NMT used a possessive determiner without the expected reflexive construction.
- Inalienable possession: All three systems failed to match the reference for “I brushed my teeth” or “You brushed your teeth.”
- Inalienable possession: PBMT correctly translated “I raised my hand,” whereas NMT produced a different lexical rendering and Google used a different tense.
- Inalienable possession: Google alone correctly translated “He turned his head”; PBMT and NMT retained possessive determiners and mistranslated or altered the construction.
- Inalienable possession: The zero-relative challenge tests whether an omitted English relative pronoun becomes an explicit French relative pronoun.
- Inalienable possession: Google correctly translated all three zero-relative examples, while PBMT and NMT failed on each example.