Source-linked AI summary
What do Neural Machine Translation Models Learn about Morphology?
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, James Glass
TL;DR
The paper addresses limited knowledge about what neural MT models learn about source and target-language morphology. It probes representations through POS and morphological tagging across architectural and linguistic factors, finding systematic differences by representation, depth, target language, and module.
Problem
Little is known about what and how much neural MT models learn about each language’s features, including word morphology and the roles of architecture, representations, and target language.
Method
The study extracts representations from trained NMT encoders and decoders, then evaluates them with classifiers on POS and morphological tagging tasks.
Results
Character-based representations better capture morphology, lower layers better capture word structure, target language affects source representations, and encoder and decoder quality differs little.
Takeaways & Limitations
NMT representations distribute information about word structure and meaning across representation types, network depth, target language, and encoder-decoder components.
Takeaways & Limitations
Future work should extend the analysis to byte-pair encoding, deeper networks, and more semantically oriented tasks.
Abstract
from arXiv · showhide
Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture. However, little is known about what these models learn about source and target languages during the training process. In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.
1 Introduction
This work asks what neural MT models learn about word structure and how architecture, representations, and target languages affect that learning. It evaluates POS and morphological tagging from extracted representations to quantify these properties.
- The study investigates which NMT components capture word structure, how layers and modules divide labor, and how representations and target languages affect morphology learning.
- A classifier trained on features extracted from a pretrained NMT model provides a quantitative proxy for the quality of representations relevant to POS and morphology.
- Character-based representations are much better for learning morphology, especially for low-frequency words, while word-based models suffice for common-word structure.The improvement for character-based models correlates with better BLEU scores.
- Lower encoder layers capture word structure better, whereas deeper networks improve translation quality and may focus more on word meaning.
- Translating into morphologically poorer languages leads to better source-side word representations, with only partial correlation to BLEU scores.
- Encoder and decoder representations have similar quality, but attention affects encoder representations more than decoder representations.
2 Methodology
The methodology freezes trained NMT models, extracts word representations, and tests them with classifiers for POS and morphological tagging. This isolates how informative different encoder and decoder representations are for word structure.
- The encoder maps a source sentence to a vector representation, after which the decoder maps that representation to the target sentence.
- After training attention-based LSTM encoder-decoders, the study extracts word-level encoder features and trains classifiers to predict POS or morphological tags.
- The same feature-extraction and classifier procedure is applied to decoder representations for target-side morphology analysis.
- A simple feed-forward classifier with one hidden layer and a ReLU non-linearity assesses representation quality rather than state-of-the-art tagging performance.
3 Data
The experiments use comparable multilingual translation and annotation data spanning morphologically varied languages. Classifiers are evaluated with both predicted and gold annotations where available.
- The language-pair experiments include Arabic-English, German-English, French-English, Czech-English, Arabic-Hebrew, and Arabic-German.
- Translation models are trained on the WIT3 TED-talk corpus using official development and test sets, with reported figures averaged over test sets.
- POS and morphological classifiers use both gold-standard and freely predicted annotations, reporting both results where gold data are available.
- A linear classifier produced similar trends but lower overall results than the nonlinear classifier.
- Table 2 reports POS accuracy for word-based and character-based representations on gold and predicted tags, alongside BLEU scores.
4 Encoder Analysis
Encoder representations capture morphology differently depending on representation type, frequency, depth, and target language. Character-based and lower-layer representations generally provide stronger morphological features, while target-language effects are not reducible to BLEU.
- 4.1 Effect of word representation: Character-based encoders generate better POS and morphological representations, especially for morphologically rich languages such as Arabic and Czech.The same pattern appears in full morphological tagging.
- 4.1 Effect of word representation: +14.3% POS and +14.5% morphology accuracy overall, with +37.6% POS and +32.7% morphology accuracy on OOV words for character-based representations.The comparison uses Arabic-Hebrew encoder features.
- 4.1 Effect of word representation: Character-based models reduce determiner-to-noun confusions by capturing Arabic’s “Al-” definite-article prefix.Word-based models more often misclassify determiner-bearing tags as non-determined nouns.
- 4.2 Effect of encoder depth: Passing word vectors through one encoder layer improves POS tagging, but first-layer representations outperform second-layer representations for word structure.The pattern holds across the reported Arabic POS and morphology experiments and is also observed in French, German, and Czech.
- 4.2 Effect of encoder depth: +1.11/+0.56 BLEU for Arabic-Hebrew word/char-based models accompanies deeper encoders, even as morphology quality decreases.This separates translation-quality gains from morphology-representation quality.
- 4.3 Effect of target language: English-target encoders produce better Arabic POS and morphology representations than German-target encoders, which outperform Hebrew-target encoders.The reported ordering occurs despite English being morphologically poorer; higher BLEU alone does not guarantee better morphological representations.
5 Decoder Analysis
Decoder representations support target-side POS and morphology tagging at quality close to encoder representations. Attention affects encoder representations substantially more than decoder representations, while character-based inputs improve tagging in both modules.
- 5 Decoder Analysis: Decoder features are extracted for target words while the decoder receives the correct target words one by one, as in NMT training.A classifier then predicts target-side POS or morphological tags from these features.
- 5 Decoder Analysis: Decoder representations show a modest drop in quality relative to encoder representations.The comparison uses Arabic-English encoder and English-Arabic decoder representations, with similar small drops in higher-quality directions.
- 5 Decoder Analysis: Removing attention significantly decreases encoder-representation quality but only mildly harms decoder-representation quality.The decoder therefore does not rely on attention to the same extent for target-word representations.
- 5 Decoder Analysis: Character-based representations perform better for POS tagging in both encoder and decoder settings.Translation BLEU differs by direction: character-based models improve Arabic-to-English translations but not English-to-Arabic translations.
6 Related Work
Prior work analyzed neural representations qualitatively or addressed morphological representation techniques in MT without directly measuring learned-representation quality. This paper uses quantitative probing to evaluate what NMT representations encode.
- 6 Related Work: Neural-network analysis has used hidden-unit visualizations, but these studies are often qualitative and anecdotal.The passage contrasts visualization with a quantitative analysis approach.
- 6 Related Work: Related studies considered representations for morphologically rich languages in MT but did not directly measure the quality of learned representations.This identifies the methodological gap addressed by the paper.
- 6 Related Work: Earlier MT systems used word segmentation, factored translation, and reordering models to represent morphological knowledge.Characters and other subword units were also used before becoming increasingly popular in neural MT.
7 Conclusion
The study evaluates neural MT representations for POS and morphological tagging and finds systematic effects from representation type, network depth, target language, and module. These findings also motivate future analyses and development choices.
- Character-based representations outperform word-based representations for learning morphology, especially in rare and unseen words.
- Lower network layers capture morphology better, whereas deeper layers improve translation performance and may focus more on word meaning.
- Translating into morphologically poorer languages produces better source-side representations, with only partial correlation to BLEU scores.
- Encoder and decoder representations have similar quality, but attention affects encoder representations more than decoder representations.
- Jointly learning translation and morphology could improve representations and translation, while requiring attention to encoding layer and word representation.
- Future work should examine byte-pair encoding, deeper networks, and more semantically oriented tasks.
A.1 Training Details
The experiments use a two-layer attention-based LSTM encoder-decoder and a character-based CNN with a highway network. POS and morphological predictions come from a one-hidden-layer feed-forward classifier.
- The POS and morphological classifier uses one hidden layer, dropout, ReLU, Softmax, and Adam optimization.
- The neural MT system is a two-layer LSTM encoder-decoder with attention, using 500-dimensional word vectors and LSTM states.
- Character-based models use a CNN with a highway network over characters, 1000 feature maps, and kernel width six.
A.2 Data and Taggers
Translation models are trained on the WIT3 TED-talks corpus, with additional gold-tag data from Arabic Treebank and the Tiger corpus. Multiple taggers annotate POS and morphological features across languages.
- Datasets: All translation models are trained on the TED-talks corpus included in WIT3.
- Datasets: Gold-tag experiments use Arabic Treebank for Arabic and the Tiger corpus for German.
- POS and morphological taggers: MADAMIRA, Tree-Tagger, LoPar, and MXPOST annotate POS or morphological tags for Arabic, Czech, French, German, and English.
- Evaluation scope: The study prioritizes analyzing what NMT components learn about morphology rather than achieving state-of-the-art tagging accuracy.
A.3 Supplementary Results
Supplementary results consistently compare layers, representations, target languages, and decoder behavior. They reinforce the reported morphology findings and show that decoder pre-Softmax representations mainly encode the current input word rather than the next word’s POS.
- Encoder results: Layer 1 outperforms layers 0 and 2, while character-based representations outperform word-based representations for morphology.
- Target-language effects: Translating into English, a morphologically poor language, yields better source representations; additional decoder results are reported separately.
- Decoder results: Decoder representations cannot reliably predict the next word’s POS tag, including without attention or with character-based representations.
- Decoder results: The pre-Softmax decoder representation contains substantial information about the current input word, while most next-word prediction occurs in the Softmax layer.