Source-linked AI summary
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
Gongbo Tang, Mathias Müller, Annette Rios, Rico Sennrich
TL;DR
The paper asks whether shorter network paths explain non-recurrent NMT performance, or whether semantic feature extraction provides an alternative explanation. It evaluates RNNs, CNNs, and Transformers on subject-verb agreement and WSD using contrastive translation pairs. Transformers and CNNs are not superior to RNNs on long-distance agreement, while Transformers excel at WSD; attention-head count also affects long-range dependency modeling.
Problem
The paper addresses the untested claim that shorter network paths make CNNs and self-attentional NMT architectures better at modeling long-range dependencies, while considering semantic feature extraction as an alternative explanation.
Method
The paper evaluates RNNS2S, ConvS2S, and Transformers on subject-verb agreement and WSD by scoring contrastive translation pairs.
Results
Transformers and CNNs show no evidence of superiority over RNNs on long-distance subject-verb agreement, while Transformers excel at WSD; attention-head count affects long-range dependency modeling.
Takeaways & Limitations
Assessing NMT architectures requires examining task-specific strengths and weaknesses rather than relying only on overall BLEU scores.
Takeaways & Limitations
The findings identify limited context size as a major problem for ConvS2S, while scale invariance may explain part of its remaining performance difference.
Abstract
from arXiv · showhide
Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.
1 Introduction
The paper questions whether shorter network paths explain non-recurrent NMT gains and tests long-range dependency modeling against semantic feature extraction. Across the proposed evaluations, Transformers excel at WSD, while neither Transformers nor CNNs show superiority over RNNs for long-distance subject-verb agreement.
- Architecture comparisons based mainly on BLEU make it difficult to attribute gains to architectural properties.
- Shorter paths have been proposed to make Transformers and CNNs better suited than RNNs to capturing long-range dependencies.The claim had not yet been empirically tested in this context.
- The paper evaluates RNNS2S, ConvS2S, and Transformers on subject-verb agreement and WSD using contrastive translation test sets.Subject-verb agreement targets long-range dependencies, whereas WSD targets semantic feature extraction.
- Transformers and CNNs show no evidence of superiority over RNNs in modeling subject-verb agreement over long distances.This result challenges the theoretical explanation based solely on shorter network paths.
- Many-headed multi-head attention is essential for modeling long-distance phenomena with only self-attention.
- Transformers excel at WSD, supporting the hypothesis that their performance is related to strong semantic feature extraction.
2 Related work
Related work compares CNNs, RNNs, and Transformers across sequence and language tasks, often revealing task-dependent strengths. This paper differs by evaluating NMT architectures beyond aggregate BLEU, using targeted tests of syntax-related long-range dependencies and semantics.
- CNNs have been reported as better than RNNs on semantics-related tasks, while RNNs perform better on syntax-related tasks, especially for longer sentences.
- RNNs have outperformed CNNs on subject-verb agreement, a proxy for capturing long-range dependencies.
- A Transformer language model performed worse than an RNN language model on subject-verb agreement, especially as subject-verb distance increased.
- CNNs were concluded to be better than RNNs for sequence modeling, but performed much worse than state-of-the-art LSTMs on some tasks.
- Transformer models surpassed RNN models on historical spelling normalization only in high-resource conditions.
- Unlike previous studies, this paper focuses on NMT architecture comparisons beyond predominantly BLEU-based evaluation.
3 Background
The paper describes RNN, CNN, and self-attentional NMT architectures, then motivates contrastive evaluation for targeted linguistic phenomena rather than overall BLEU quality.
- NMT architectures: All three NMT architectures use encoder–decoder models whose components interact through soft attention.The evaluated families are RNN-based models, CNN-based models, and Transformer-based models.
- RNN-based NMT: RNNs connect each token state directly only to the previous state, so tokens separated by n positions have path length n.RNNs are stateful networks that update as new inputs arrive.
- CNN-based NMT: CNNs capture local correlations hierarchically, with context determined by kernel size and depth; a 2-layer kernel-3 CNN connects the first and fifth tokens in 2 convolutions.The largest context size is L(k−1), while local-token path length is ⌈n/(k−1)⌉.
- Transformer-based NMT: Self-attention directly connects every token to every other token, giving the first and fifth tokens a path length of 1.Transformers also use multiple attention heads and positional embeddings.
- Targeted evaluation: Contrastive evaluation pairs a human reference with automatically corrupted variants and checks whether the model scores the correct translation above every contrastive alternative.Accuracy is the percentage of cases where the correct target scores higher than all contrastive variants.
- Targeted evaluation: ContraWSD tests word-sense selection by replacing a correct translation with another incorrect meaning, such as replacing line with snake for German Schlange.Its datasets contain 7,200 German→English and 6,700 German→French lexical ambiguities, with average contrastive counts of 3.5 and 2.2 respectively.
4 Subject-verb Agreement
The subject-verb agreement evaluation tests whether shorter network paths make CNNs and Transformers better at long-range dependencies than RNNs. Across controlled comparisons, the results do not support that conclusion, while Transformer performance depends substantially on attention-head count.
- Task and setup: The subject-verb agreement task evaluates NMT architectures’ ability to model dependencies between subjects and verbs separated by more than 10 words.The study reports average accuracy on instances where subject–verb distance exceeds 10 words.
- Task and setup: RNN, CNN, and Transformer models were compared under shared toolkits, hyperparameters, and training techniques to create a level playing field.The main models used Sockeye, while an additional bi-deep RNN was evaluated from Marian for robustness.
- Overall results: Transformer achieved the highest average long-range accuracy, but the results do not show that Transformers are particularly stronger than RNNs over long distances.Transformer was better at distances 11–12, whereas RNN-bideep was equally good or better at distance 13 and above.
- CNNs: ConvS2S performed substantially worse than RNNS2S for subject-verb agreement, including within the CNN’s local context size.Increasing the CNN local context improved accuracy, but ConvS2S still did not match RNNS2S.
- Overall results: BLEU scores did not correlate well with targeted long-range dependency accuracy, despite Transformers achieving much better BLEU scores than the RNN models.This may reflect BLEU’s n-gram locality and trade-offs among modeled phenomena.
- RNNs vs. Transformer: Many-headed multi-head attention was essential for modeling long-distance phenomena with self-attention, although its effect on BLEU was small.Reducing the number of heads made Transformer performance clearly worse, while increasing it substantially improved long-distance subject-verb accuracy.
5 WSD
The WSD experiments compare RNNs, CNNs, and Transformers on German-to-English and German-to-French contrastive translation tasks. Transformers strongly outperform the other architectures, while a Transformer encoder paired with an RNN decoder performs between the pure Transformer and RNNS2S.
- Overall Results: Transformers strongly outperform RNNS2S and ConvS2S on WSD for both DE→EN and DE→FR.The Transformer model on DE→EN also exceeds uedin-wmt17 in WSD accuracy despite a 1.4 lower BLEU score on newstest2017.
- Overall Results: ConvS2S has 1.7% lower WSD accuracy than RNNS2S for DE→EN, but slightly better BLEU and accuracy for DE→FR.
- Overall Results: The Transformer model leads the other architectures on WSD by 4–8 percentage points.
- Hybrid Encoder-Decoder Model: TransRNN performs better than RNNS2S but worse than the pure Transformer on both BLEU and WSD accuracy.This indicates that the decoder, as well as the encoder, affects WSD performance.
6 Post-publication Experiments
Post-publication experiments test whether implementation details and Transformer-specific components explain the original findings. They confirm that non-recurrent models are not superior to RNNs on long-distance agreement, while Transformer-specific choices contribute to—but do not eliminate—their WSD advantage.
- Post-publication robustness: The post-publication experiments were conducted to test the robustness of the paper’s claims across configurations and implementations.
- Post-publication robustness: The pre-trained 15-layer Fairseq CNN has higher BLEU and long-range-dependency accuracy than the 8-layer Sockeye CNN, but still trails RNNS2S and Transformer on subject-verb agreement.
- Minimally different models: The retrained models minimize architectural differences by applying advanced Transformer techniques to RNN and CNN models with six-layer encoders and decoders.
- Minimally different models: Transformer-specific configurations substantially improve RNNS2S and ConvS2S BLEU and perplexity, but have relatively small effects on long-distance subject-verb agreement.
- Minimally different models: Retrained Transformers still perform best on WSD, although their performance gap over the other models narrows across BLEU, perplexity, and WSD accuracy.
7 Conclusion
The paper evaluates RNNS2S, ConvS2S, and Transformers on subject-verb agreement and WSD using contrastive translation pairs. It finds no Transformer or CNN advantage over RNNs for long-distance agreement, but a clear Transformer advantage on WSD, with attention-head count affecting long-range modeling.
- Conclusion: The evaluation scores three NMT architectures on subject-verb agreement and WSD using contrastive translation pairs.
- Conclusion: There is no evidence that CNNs and Transformers are superior to RNNs in modeling subject-verb agreement over long distances.
- Conclusion: The number of heads in multi-head attention affects a Transformer’s ability to model long-range dependencies in subject-verb agreement.
- Conclusion: Transformer models excel at WSD compared with the tested CNN and RNN architectures.
- Conclusion: The findings favor assessing architecture-specific trade-offs rather than relying only on overall BLEU scores.The paper identifies combining recurrent and self-attentional models as an avenue for further research.