PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-Invariant Testing for Machine Translation

Pinjia He, Clara Meister, Zhendong Su

arXiv:1907.08710v3cs.SEcs.CL

TL;DR

Machine-translation robustness is difficult to assess, despite the potential consequences of inferior translations. SIT tests translations of similar source sentences by comparing their syntax structures. On 200 source sentences, it found 64 issues in Google Translate and 70 in Bing Microsoft Translator, with 69.5% and 70% top-1 accuracy, respectively.

  • Problem

    Assessing machine-translation robustness is difficult, while inferior translations can lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts.

    Source: §1 · p. 1

  • Method

    SIT generates similar source sentences through contextual word replacement, translates them, represents target syntax with parse trees, and flags large structural differences.

    Source: §3 · p. 3, §3.1 · p. 4

  • Results

    With 200 source sentences, SIT found 64 buggy issues in Google Translate and 70 in Bing Microsoft Translator, with 69.5% and 70% top-1 accuracy, respectively.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    SIT uncovered diverse translation errors that state-of-the-art approaches could not find, including under-translation, over-translation, incorrect modification, mistranslation, and unclear logic.

    Source: §4.4.2 · p. 7, §4.4.3 · p. 8, §4.4.5 · p. 8

    SIT can produce false positives when generated variants have strange meanings or syntax parsers produce incorrect structures.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intractability of neural machine translation (NMT) models that power modern machine translation make the robustness of these systems difficult to even assess, much less guarantee. Machine translation systems can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation systems is very difficult and has, therefore, been much under-explored. To tackle this challenge, we introduce structure-invariant testing (SIT), a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of "similar" source sentences should typically exhibit similar sentence structures. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words; (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing); (3) reports sentence pairs whose structures differ quantitatively by more than some threshold. To evaluate SIT, we use it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy issues with 69.5\% and 70\% top-1 accuracy, respectively. The translation errors are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic.

1 INTRODUCTION

Machine translation is widely used, but evaluating its robustness is difficult because neural systems produce complex outputs and common testing approaches can miss translation errors. Structure-invariant testing (SIT) compares translations of similar source sentences; tests on Google Translate and Bing Microsoft Translator found buggy issues with reported top-1 accuracy of 69.5% and 70%, respectively.

1 INTRODUCTION

  • Translation failures can lead to misunderstanding, financial loss, medical misdiagnoses, threats to personal safety, or political conflicts.The paper notes that NMT systems can also be fooled by adversarial examples or natural input noise.Source: §1 · p. 1
  • Existing machine-translation testing can miss errors because traditional code-based tests fit neural systems poorly, while common AI tests are not designed for complex translated sentences.Such AI testing can focus on illegal inputs even though inferior translations may arise from syntactically correct inputs.Source: §1 · p. 1
  • BLEU-style score thresholds can overlook grammatical errors and require manually constructed bilingual sentence pairs, making testing real-world user input labor-intensive.The paper motivates an automated method that can detect translation errors without requiring reference translations.Source: §1 · p. 2
  • SIT generates similar source sentences by changing one word, translates them, compares their syntax structures, and flags pairs with large structural differences as potential errors.The approach uses BERT [19] for sentence generation and constituency or dependency parse trees for structure representation.Source: §1 · p. 2
  • 64 Google Translate issues and 70 Bing Microsoft Translator issues were found with 69.5% and 70% top-1 accuracy, respectively, using 200 source sentences.The reported errors included under-translation, over-translation, incorrect modification, word or phrase mistranslation, and unclear logic; BLEU and ROUGE detected none of them.Source: §1 · p. 2
  • The reported test results reflect snapshots of Google Translate and Bing Microsoft Translator at testing time, and some reported errors were later addressed.The results dataset was released in July 2019.Source: §1 · p. 2

2 A REAL-WORLD EXAMPLE

A mistranslation of a German zoo admission rule led a family to misunderstand when children could enter for free. The example illustrates how translation errors can have concrete consequences beyond an awkward sentence.

2 A REAL-WORLD EXAMPLE

  • A Google Translate error omitted the birthday condition in a German zoo admission rule, and David was denied free entry despite having valid identification.The correct translation specified that children up to 15 receive free admission on their birthday when presenting valid ID.Source: §2 · p. 2
  • The zoo incident shows how an incorrect translation can cause a confusing, unpleasant experience, while other cited cases involve more serious consequences.The paper gives the example of a Facebook translation that rendered “good morning” as “attack them” in Hebrew.Source: §2 · pp. 2–3
  • SIT’s motivating example compares translations of similar English sentences and flags a potential error when the modified sentence’s target structure differs substantially.The illustrated source variants change one word, such as “smart” to “tall.”Source: §2 · p. 3

3 APPROACH AND IMPLEMENTATION

SIT takes unlabeled monolingual sentences, generates similar variants, obtains translations, represents their syntax, and reports suspicious structural differences. Its implementation uses contextual word replacement, syntax parsers, distance metrics, and a configurable threshold.

3 APPROACH AND IMPLEMENTATION

  • SIT takes unlabeled monolingual sentences and returns suspicious issues containing the original sentence and translation plus the farthest generated variants and their translations.For each original sentence, it reports either no issue or one issue if at least one generated translation is considered buggy.Source: §3 · p. 3

3.1 Generating Similar Sentences

  • Similar sentences are generated by replacing one noun or adjective with a same-part-of-speech alternative, using contextual prediction to produce plausible variants.BERT [19] supplies the masked-language-model implementation; unlike context-independent word embeddings, the masked model uses surrounding words to propose replacements.Source: §3.1 · p. 4

3.2 Collecting Target Sentences

  • The original and generated source sentences are sent to the machine-translation system under test, and their returned translations become the target sentences for analysis.The implementation obtains results through Google and Bing APIs.Source: §3.2 · p. 4

3.3 Representations of the Target Sentences

  • Syntax parsing provides representations of target-sentence structure that can support more meaningful comparisons than raw strings alone.Raw strings can appear quite different when equivalent phrases move, whereas syntax relations can preserve their structural representation.Source: §3.3 · p. 4

3.3.1 Raw Target Sentence.

  • Raw-string comparison is a baseline that can flag token changes, but sentence reordering can make equivalent translations look far apart.The Levenshtein metric counts the minimum character insertions, deletions, and substitutions needed to transform one string into another.Source: §3.3.1 · pp. 4–5, §3.4.1 · p. 5

3.3.2 Constituency Parse Tree.

  • Constituency parsing represents phrase structure as an ordered tree, and SIT compares counts of each constituency-relation type across translations.This relation-count heuristic is motivated by the expectation that phrase types remain stable when one same-part-of-speech token changes.Source: §3.3.2 · p. 5, §3.4.2 · p. 5

3.3.3 Dependency Parse Tree.

  • Dependency parsing represents direct relationships between words, and SIT compares counts of each dependency-relation type across translations.The implementation uses Stanford CoreNLP neural dependency parsers with Universal Dependencies.Source: §3.3.3 · p. 5, §3.4.3 · p. 5

3.4 Translation Error Detection via Structure Comparison

  • SIT can measure structural variation with string edit distance or constituency- and dependency-relation set differences, then retain variants beyond a chosen threshold.It reports the top-k farthest modified translations; the threshold is a manual parameter that lets users trade off false-positive and false-negative reports.Source: §3.4 · p. 5, §3.4.4 · pp. 5–6

4 EVALUATION

The evaluation tests SIT on unlabeled Web sentences from politics and business articles, with researchers manually checking reported issues. The dataset contains 200 randomly selected sentences, with long sentences excluded.

4 EVALUATION

  • The evaluation asks whether SIT finds buggy translations, what error types it detects, how efficiently it runs, and how to choose its distance threshold.Researchers manually inspected reported issues to determine whether they were buggy and classify their errors.Source: §4 · p. 6, §4.1 · p. 6

4.2 Dataset

  • The test inputs were real-world source sentences crawled from CNN politics and business articles to assess performance across different semantic contexts.For each category, the authors collected the 10 latest articles and extracted their main text.Source: §4.2 · p. 6
  • The dataset comprised 200 randomly selected sentences, 100 per category, and excluded sentences longer than 35 words.The authors chose relatively short sentences to demonstrate that translation systems can return inferior results even for simple inputs.Source: §4.2 · p. 6

4.3 The Effectiveness of SIT

The evaluation measures how accurately SIT flags translation issues and how many buggy sentences it finds. Dependency-based SIT performs best on top-1 accuracy, while structure-based variants outperform the raw-sentence variant.

4.3.2 Results.

  • Table 2 reports that SIT (Dep) finds 64 Google Translate and 70 Bing Microsoft Translator buggy issues, with 69.5% and 71% top-1 accuracy, respectively.Source: Table 2 · p. 6, §4.3.2 · p. 7

4.3.1 Evaluation Metric.

  • Top-k accuracy counts an issue as accurate when an error occurs in a reported generated sentence or the original sentence, divided by all returned issues.The metric evaluates both the accuracy of reported issues and the number of buggy sentences found.Source: §4.3.1 · p. 6

4.3.2 Results.

  • SIT (Con) and SIT (Dep) consistently outperform SIT (Raw), while the two structure-based variants have comparable accuracy and buggy-issue counts.Source: §4.3.2 · pp. 6–7
  • SIT (Dep) identifies 79 Google Translate and 66 Bing Microsoft Translator unique errors only in generated sentence pairs, beyond 55 and 60 errors in original translations.The generated-pair errors were not found in the original translations.Source: §4.3.2 · p. 7, Table 3 · p. 7
  • The authors do not empirically compare SIT with [96] because it targets under- and over-translation, or because its input requirements do not fit these datasets.Source: §4.3.2 · p. 7

4.4 Translation Error Reported by SIT

SIT identifies five kinds of translation errors, with under-translation, word/phrase mistranslation, and unclear logic accounting for most reported errors. Examples show that errors can range from omitted or extra words to incorrect semantic relationships, and multiple kinds can occur together.

4.4 Translation Error Reported by SIT

  • Under-translation, word/phrase mistranslation, and unclear logic account for most errors reported by SIT.The five error types are presented as subsets of lexical, syntactic, or semantic errors.Source: §4.4 · p. 7

4.4.1 Under-Translation.

  • An omitted source phrase can change meaning: leaving out “to Congress” turns illegal lying into merely inappropriate behavior.The paper identifies this as under-translation.Source: §4.4.1 · p. 7

4.4.2 Over-Translation.

  • Over-translation adds target words absent from the source or repeats translations; it can co-occur with under-translation.The example includes an untranslated source phrase alongside an extra target word.Source: §4.4.2 · p. 7

4.4.3 Incorrect Modification.

  • Incorrect modification assigns a modifier to the wrong element, as when Google Translate attaches “new” to “hub” rather than “auto manufacturing.”The authors suggest an “A of B” pattern in training data may contribute to another such error.Source: §4.4.3 · p. 8

4.4.4 Word/phrase Mistranslation.

  • Word/phrase mistranslation includes polysemy errors, such as interpreting “admit” as agreeing unwillingly, and wrong translations, such as reading “South” as “South Korea.”The paper gives these as two sub-categories: ambiguity of polysemy and wrong translation.Source: §4.4.4 · p. 8

4.4.5 Unclear Logic.

  • Unclear logic occurs when individual tokens are translated correctly but their relationship is wrong, as in rendering “serving the country in elected office” as serving in office “as a country.”The authors connect this error to failure to understand the logical relation between the phrases.Source: §4.4.5 · p. 8

4.4.6 Sentences with Multiple Translation Errors.

  • One sentence pair can contain several error types: the example combines mistranslation, an extra target word, and an incorrect subject interpretation.In the example, “covering” is rendered as “holding,” “church” is added, and the subject is misread as attending rather than covering a funeral train.Source: §4.4.6 · p. 8

4.5 The Running Time of SIT

SIT’s non-translation steps run in about one to two minutes, with raw-sentence and dependency-based representations faster than constituency parsing. Translation-result collection takes most of the reported time, while dependency parsing adds little time for the tested workload.

4.5 The Running Time of SIT

  • 19 seconds to parse 2000+ sentences makes SIT (Dep) comparable in running time to SIT (Raw), which requires no structure-generation time.The authors characterize the dependency-parsing cost as efficient and reasonable.Source: §4.5 · p. 9
  • SIT spends most of its time collecting translations through the Google and Bing APIs, and those translation-time values are reused across implementations.In practice, the authors expect this step to take much less time when testing users’ own software.Source: §4.5 · p. 9
  • Around 1 min for SIT (Raw) and SIT (Dep), versus around 2 mins for SIT (Con), shows the non-translation steps are efficient for offline testing.The reported running times are averages across 10 runs; SIT (Con) is slower because constituency parsers are slower than dependency parsers.Source: §4.5 · p. 9, Table 5 · p. 9

4.6 The Impact of Distance Threshold

The distance threshold controls how many buggy issues SIT reports and their top-1 accuracy. Small thresholds favor finding more issues, while large thresholds favor higher accuracy; on the tested datasets, both measures are stable at threshold extremes and fluctuate in the middle.

4.6 The Impact of Distance Threshold

  • The threshold controls both buggy-issue count and top-1 accuracy; lowering it is expected to increase reports while reducing accuracy.The figure studies SIT (Dep) on Bing Microsoft Translator using the Politics and Business datasets.Source: §4.6 · p. 9, Figure 13 · p. 9
  • On the tested datasets, issue counts and top-1 accuracy remain stable at small or large thresholds but fluctuate in the middle.The authors report a similar threshold impact for Google Translate.Source: §4.6 · p. 9, Figure 13 · p. 9
  • A small threshold reports the most issues with decent accuracy, while a large threshold reports fewer issues with very high accuracy.The guidance gives a small dependency-set threshold of 4 with 70% top-1 accuracy, and a large threshold of 15 with 86% top-1 accuracy.Source: §4.6 · p. 9

4.7 Fine-tuning with Errors Reported by SIT

The authors test whether SIT-reported sentences can improve an NMT model through fine-tuning. After fine-tuning on the 105 reported sentences, the model translated all of them correctly.

4.7 Fine-tuning with Errors Reported by SIT

  • The experiment used a global-attention encoder-decoder with single-layer unidirectional LSTMs, trained on 2M CWMT sentence pairs.Google and Bing models were not used because they are not open-source.Source: §4.7 · pp. 9–10
  • All 105 SIT-reported sentences were translated correctly after 15 fine-tuning epochs, and validation BLEU increased by 0.13.The authors say the validation gain shows, to some degree, that other sentences also improved.Source: §4.7 · p. 10
  • SIT found 105 buggy sentences from 40 English validation sentences, which were labeled with correct translations and used for fine-tuning.Inputs were selected from WMT’17 validation data, limited to sentences of at most 12 words with all words in the model vocabulary.Source: §4.7 · p. 10

5 DISCUSSIONS

SIT can produce false positives when generated sentences have odd meanings, parsers err, or valid translations differ structurally. The authors describe using reported cases for bug fixing, fine-tuning, debugging, and future model design.

5.1 False Positives

  • False positives can arise from semantically strange substitutions, parser errors, or multiple correct translations with different structures.SIT ranks the top-k suspicious pairs by distance from the original target sentence to reduce the impact of these factors.Source: §5.1 · p. 10

5.2 Building Robust Translation Software

  • Reported mistranslations can serve as early alarms that let developers hard-code translation mappings as a quick bug-fixing option.The authors also identify the reported sentences as a possible fine-tuning set.Source: §5.2 · p. 10
  • Because each reported sentence pair differs by one word, developers can use the pairs for further analysis or debugging.The authors compare this to input minimization or localization in traditional software debugging.Source: §5.2 · p. 10
  • The authors propose structural invariance as a possible inductive bias for designing robust NMT models and identify it as future work.They note that debugging and fixing NMT software is harder because its logic mainly lies in model structure and parameters.Source: §5.2 · p. 10

6 RELATED WORK

The related work situates SIT among AI and NLP robustness testing, machine translation research, and metamorphic testing. It distinguishes SIT through its black-box focus, broader error coverage, and techniques for generating similar sentences and comparing parsed structures.

6.1 Robustness of AI Software

  • Unlike many AI robustness approaches that require gradients or activation values, SIT is black-box and focuses on machine translation systems.The cited approaches address testing, debugging, online adversarial-example detection, or robust training.Source: §6.1 · p. 10

6.2 Robustness of NLP Algorithms

  • Prior NLP robustness work focused on simpler tasks such as text classification, while MT4MT was limited to simple subject-verb-object sentences and name or brand substitutions.The authors describe SIT as more systematic and not limited to specific translation errors, with BERT-based substitution and syntax parsers as its realization.Source: §6.2 · pp. 10–11

6.3 Machine Translation

  • Where cited NMT architecture work aims to improve model capability, this paper focuses on robustness, which the authors say complements accuracy research.The cited architectures include recurrent, convolutional, and attention-only networks.Source: §6.3 · p. 11

6.4 Metamorphic Testing

  • SIT applies metamorphic testing to machine translation by generating cases from existing ones and checking domain-specific relations across outputs.The paper identifies structural invariance as its metamorphic testing approach.Source: §6.4 · p. 11

7 CONCLUSION

The conclusion presents SIT as a general approach to testing machine translation and reports successful tests of Google Translate and Bing Microsoft Translator. The authors frame it as an initial step toward systematic testing and identify extending and refining the approach as future work.

7 CONCLUSION

  • 64 Google Translate and 70 Bing Microsoft Translator buggy issues were found, with top-1 accuracy of 69.5% and 70%, respectively.The authors say SIT uncovers diverse translation errors that state-of-the-art approaches cannot find.Source: §7 · p. 11

Related papers

Loading 1907.08710v3…