Source-linked AI summary
Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation
Leshem Choshen, Omri Abend
TL;DR
Long-distance dependencies remain difficult for machine translation, raising the question of whether they are still a challenge for Transformers. The paper develops automatically extracted challenge sets to evaluate them and finds that removing locality bias does not remove the difficulty. These sets offer a complementary evaluation perspective, though automatic extraction can be noisy and may not select representative examples.
Problem
Long-distance dependencies are difficult to detect and can require non-monotonic translation, while coarse metrics may not reflect performance on these infrequent phenomena.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Targeted challenge sets provide a complementary perspective on MT performance and enable scalable evaluation of important linguistic phenomena.
Source: §1 · p. 2, §5 · p. 9Automatic extraction can be noisy and may select sentences that are not representative of each phenomenon.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
We show that the state of the art Transformer Machine Translation (MT) model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model. Since most dependencies are short-distance, common evaluation metrics will be little influenced by how well systems perform on them. We, therefore, propose an automatic approach for extracting challenge sets replete with long-distance dependencies and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena.
1 Introduction
Long-distance dependencies remain difficult for NMT, and the paper proposes automatically extracted challenge sets to evaluate them. Its findings distinguish the Transformer’s locality bias from its continuing difficulty with these dependencies.
1 Introduction
- Long-distance dependencies remain an open question for MT because they are harder to detect and can require non-monotonic translation.Source: §1 · p. 1
- The method extracts reordering and lexical long-distance dependencies, using a dependency parser to find linguistic phenomena in parallel-corpus source text.The test sets cover German–English and English–German, and the approach can scale to other languages with a sufficiently good parser.Source: §1 · pp. 1–2
- Self-attention Transformers with learned positional embeddings show no locality bias, yet long-distance dependencies remain challenging and lexical dependencies grow harder with distance.The authors also report that the default Transformer implementation retains some locality bias, which learned positional embeddings can relieve.Source: §1 · pp. 1–2
- Because such dependencies are too infrequent to substantially affect coarse-grained metrics such as BLEU or TER, targeted challenge sets provide a complementary view of system performance.Source: §1 · p. 2
2 Related Work
Prior work describes how recurrent and attention-based architectures handle dependencies, while noting that long-distance phenomena remain difficult. The section also motivates larger, interpretable challenge sets as a complement to coarse evaluation metrics.
2.1 Long-distance Dependencies in MT
- LSTM systems must preserve information across intermediate steps, and their performance decreases with trigger-to-prediction distance despite evidence that they can learn structural information.The Transformer instead makes input positions equally reachable through self-attention and represents position with learned or sine embeddings.Source: §2.1 · p. 2
- Most dependencies are short—62% in UD EWT are at most one word apart—but distant phenomena such as subject–verb agreement can challenge locality-biased models.Earlier MT approaches used locality as a useful bias, while syntax-based systems also have limits with arbitrarily different word orders.Source: §2.1 · pp. 2–3
2.2 MT Evaluation
- Earlier challenge sets were manually compiled and small, limiting evaluation to manual assessment of a few examples per phenomenon.String-similarity metrics are common but partial and coarse-grained, motivating interpretable phenomenon-specific evaluation.Source: §2.2 · p. 3
- Parser-based extraction produces larger challenge sets that support standard MT measures per phenomenon and can accommodate future evaluation advances.Source: §2.2 · p. 3
3 Locality in SoTA NMT
The section tests whether source-token order affects MT systems using controlled permutations. Learned-position Transformers show no locality bias in this test, unlike BiLSTMs and Transformers with sine positional embeddings.
3 Locality in SoTA NMT
- The tested BiLSTM exhibits locality bias, whereas the self-attention Transformer with learned positional embeddings does not.Source: §3 · p. 3
3.1 Methodology
- The controlled test compares regular-order data with data using a fixed source-token permutation, holding target data constant.The German–English experiment uses fixed-length 18-token sentences from WMT2015, with 1,000 held out for testing.Source: §3.1 · p. 4
3.2 Results
- Nematus and the Transformer with SinePEs lose performance on permuted input, while learnedPE Transformer BLEU scores remain essentially the same across tested settings.Source: Table 1 · p. 4, §3.2 · p. 4
3.3 Discussion
- With learned positional embeddings that do not introduce locality bias, globally applicable information can be added at any fixed position in a Transformer input sequence.BiLSTMs often require the same information at each input token for effective use.Source: §3.3 · pp. 4–5
4 LDD Challenge Sets
The section presents an automatic approach to evaluating long-distance dependencies (LDD) and tests it on English–German and German–English. Challenge sets reveal difficulties across models, with lexical LDD becoming harder as dependency distance increases.
4.1 Methodology
- The method detects reordering LDD through aligned-word distance and lexical LDD through source-side dependency rules, filtering for sufficiently distant words.The authors use syntactic parses to find challenging phenomena, not to assume MT systems internally represent those parses; automatic extraction permits sets of hundreds or thousands of pairs and standard evaluation measures.Source: §4.1 · p. 5
- Automatic extraction can be noisy and may select sentences that are not representative of a phenomenon or that do not pose a real translation challenge.For example, the fixed pairing of befinden with sich may let a system ignore the reflexive pronoun.Source: §4.1 · p. 6
4.2 A Test Case on Extracting Sets
- The English–German and German–English test case extracts reordering and lexical-LDD sets from News and Books, including reflexive verbs, verb-particle constructions, and preposition stranding.Manual checks found the target phenomenon in 85% of German sentences, 87% of English News sentences, and 86% of Books sentences.Source: §4.2 · p. 6, §4.3 · pp. 7–8
4.3 Experiments
- Challenge subsets score below full-corpus baselines across models and domains; reordering sets also score lower on RIBES.Table 4 reports BLEU comparisons, while Table 6 reports RIBES for reordering LDD.Source: Table 4 · p. 7, §4.3 · p. 8, Table 6 · p. 8
- Lexical-LDD performance decreases with dependency distance; manual German–English annotation finds 60%, 54%, and 38% correct translations at d=1, 2, and 5, respectively.The manual results replicate the distance-related decrease observed in the challenge-set evaluation.Source: §4.3 · p. 8, Table 5 · p. 8
- Length-matched controls suggest sentence length is unlikely to be the only cause of the observed trends; the authors point to nonstandard linguistic structure and varying distances as challenges for Transformers.Challenge-set BLEU was lower than that of randomly sampled, similarly length-distributed corpora in all German-to-English cases; English-to-German trends were similar but less pronounced.Source: §4.3 · pp. 8–9
5 Conclusion
The paper finds that Transformers can be made indifferent to absolute word order, but this does not mean they overcome long-distance dependencies in naturalistic data. It also shows that parser-based challenge sets can support scalable evaluation of difficult phenomena.
5 Conclusion
- Using a UD parser, the authors show that challenge sets can be extracted at a scale suitable for evaluating important and challenging phenomena.They argue that targeted evaluation provides a detailed, complementary perspective because these phenomena may be too infrequent to affect coarse-grained measures such as BLEU or TER.Source: §5 · p. 9
- Transformers can be made indifferent to absolute word order, yet still struggle with long-distance dependencies in naturalistic data.The difficulty of inducing linguistic structure in evasive cases motivates exploring more linguistically informed models.Source: §5 · p. 9
A Manual Evaluation of Extraction Procedure
Manual validation finds high precision for several extracted phenomena, with differences across languages and types. German reflexive-verb candidates also include non-reflexive pronouns that pose a disambiguation challenge.
A Manual Evaluation of Extraction Procedure
- 96% of German phrasal-verb candidates indeed contain a phrasal-verb long-distance dependency.For German reflexive-verb candidates, 63% include reflexive verbs at distance at least 1; two thirds of the remaining cases (25%) include non-reflexive verbal pronouns.Source: A Manual Evaluation of Extraction… · p. 13
- Some German non-reflexive verbal pronouns are not lexical long-distance dependencies, but still require disambiguation from reflexive verbs.The paper reports that these extracted non-reflexive cases show similar trends to reflexive cases in the following analysis.Source: A Manual Evaluation of Extraction… · p. 13
- English detection precision is high for reflexive and phrasal verbs but lower for preposition stranding.Most wrongly extracted preposition-stranding examples involve elided or difficult-to-detect prepositional objects, which the authors consider sufficiently similar in the difficulties they pose.Source: A Manual Evaluation of Extraction… · p. 13
B Manual MT Performance evaluation
Manual Transformer evaluation finds that translation performance declines as dependency distance increases, including across reflexive and non-reflexive pronoun categories. The analysis uses annotated German-to-English examples and reports results by phenomenon and distance.
B Manual MT Performance evaluation
- For German-to-English evaluation, an annotator marked reference counterparts for relevant German tokens, then two English-proficient annotators judged whether the Transformer conveyed the marked meaning.The analysis used the same sample of 180 German sentences as the German extraction-detection evaluation; inter-annotator agreement was κ = 0.79.Source: B Manual MT Performance evaluation · p. 13
- Transformer performance decreases with distance d across reflexive and non-reflexive pronoun categories.The decline is smaller between d = 1 and d = 2 for reflexive verbs; Tables 8 and 9 report accuracy by distance for the overall phenomena and reflexive subtypes.Source: B Manual MT Performance evaluation · p. 13, Table 8 · p. 13, Table 9 · p. 14
- Table 7 reports the ratio of extracted English sentences that contain the target lexical long-distance dependency, organized by phenomenon and corpus.Source: Table 7 · p. 13