Source-linked AI summary
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, James Glass
TL;DR
The paper asks what linguistic information NMT encoders learn and how that information varies across layers and target languages. It extracts layer-specific encoder features to classify POS and semantic tags, using classifier performance to assess representation quality. Higher layers are better for semantics, lower layers tend to be better for POS, and target-language effects on source representations are small, especially for higher-quality NMT models.
Problem
It is unclear what linguistic information NMT models learn internally, particularly beyond the morphological and syntactic properties studied in earlier work.
Method
The authors train NMT systems on parallel data, extract encoder representations from different layers, and use them as features for POS and semantic-tagging classifiers.
Results
Higher encoder layers better capture semantics while lower layers tend to perform better for POS, and target-language effects on source representations are small.
Takeaways & Limitations
Layer depth provides distinct linguistic information: higher layers are useful for semantic properties, while lower layers are useful for POS properties.
Takeaways & Limitations
The study is limited to word-level POS and semantic tagging, with future work proposed for relational tasks such as dependency relations or predicate-argument structure.
Abstract
from arXiv · showhide
While neural machine translation (NMT) models provide improved translation quality in an elegant, end-to-end framework, it is less clear what they learn about language. Recent work has started evaluating the quality of vector representations learned by NMT models on morphological and syntactic tasks. In this paper, we investigate the representations learned at different layers of NMT encoders. We train NMT systems on parallel data and use the trained models to extract features for training a classifier on two tasks: part-of-speech and semantic tagging. We then measure the performance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding representation learning in NMT models. For instance, we find that higher layers are better at learning semantics while lower layers tend to be better for part-of-speech tagging. We also observe little effect of the target language on source-side representations, especially with higher quality NMT models.
1 Introduction
The paper examines what linguistic information NMT encoders learn across layers, focusing on POS and semantic tagging. It finds lower layers more useful for POS, higher layers better for semantics, and little target-language effect on source representations.
- NMT models improve translation quality, but their learned linguistic representations remain difficult to interpret.
- The study evaluates semantic tagging alongside POS tagging to investigate how NMT representations encode semantics across encoder layers.Semantic tagging captures distinctions that ordinary POS tags can miss.
- Semantic distinctions can affect translation choices, requiring different representations for the same POS form in different contexts.For example, “herself” maps to different Spanish forms depending on whether it is reflexive or emphatic.
- Lower-layer representations are usually better for POS, whereas higher-layer encoder representations better capture semantics, especially discourse functions and noun concepts.
- Target language has little effect on source-side representations, and this effect diminishes as NMT training data increases.
2 Methodology
The methodology treats a trained NMT encoder as a feature generator and evaluates its layer-specific representations through word-level POS and semantic tag classifiers. Classifier performance provides a quantitative comparison of representation quality across layers and target-language models.
- The authors train an NMT system on parallel data, freeze it, and use encoder outputs as features for classification.
- For each word, a classifier receives the output of a selected encoder layer and predicts its linguistic label.
- Classifier performance serves as a proxy for the quality of the NMT representation for the evaluated task.
- The experiments compare POS and SEM classifiers using English encoder features from NMT systems trained with several target languages.
3 Data and Experimental Setup
The experiments use multilingual parallel data and supervised POS and semantic-tagging datasets, with NMT encoder representations evaluated by a controlled feedforward classifier. Baselines include frequent tags and unsupervised embeddings, alongside a Word2Tag upper bound.
- Data: NMT models are trained on English-to-other-language data from the multilingual United Nations corpus, enabling comparable cross-linguistic analysis.The corpus contains six languages and 11 million multiparallel sentences; models use the first 2 million training sentences.
- Data: SEM tags distinguish contextual meanings that POS tags often collapse, including demonstrative, named-entity, and discourse-function differences.
- Data: Semantic tagging assigns 66 fine-grained tags grouped into 13 coarse categories, while the Penn Treebank POS dataset contains 34 tags.
- Experimental setup: The NMT systems use attentional four-layer LSTM encoder-decoder models, and the classifier is a one-hidden-layer feedforward network without separate context access.
- Experimental setup: The evaluation compares NMT features with most-frequent-tag and unsupervised-embedding baselines and a Word2Tag upper bound.
4 Results
Across NMT representations, lower layers generally favor POS tagging while higher layers better capture semantic information. These patterns persist across architectural variants, although target-language effects are small and some semantic classes remain exceptions.
- POS tagging: NMT representations reach around 91–92% POS accuracy above layer 0, matching the MFT baseline but remaining below the Word2Tag upper bound.This improvement over unsupervised embeddings does not exceed the stronger most-frequent-tag baseline.
- POS tagging: POS accuracy peaks at layer 1 in 3/5 target languages, while higher layers do not consistently improve performance.The three languages are Arabic, Russian, and Chinese; Spanish and French instead perform better at layer 4.
- SEM tagging: Semantic tagging reaches around 87–88% with layers 1–4, far above the UnsupEmb and MFT baselines but below the Word2Tag oracle.The results indicate that NMT representations contain useful semantic information.
- SEM tagging: Layer 4 significantly improves SEM tagging over layer 1 for every target language except Chinese, with p < 0.001 for the reported layer comparisons.Layers 2 and 3 do not consistently improve semantic tagging, whereas the layer-1-to-layer-4 difference is significant for all languages except Chinese.
- Architectural variants: English autoencoders produce poor POS and SEM representations despite achieving 96.6 BLEU for sentence recreation, especially at higher layers.Layer-4 representations are around 5% below those from machine-translation models.
- Target language: Across target languages, POS and SEM representation differences are usually about 0.5%, with Chinese producing slightly worse representations.The smaller target-language effect than previously reported is attributed to the larger training corpora used here.
- Semantic tag analysis: Higher layers better capture discourse relations, noun semantics, events, logic relations, and comparative constructions, while modality favors layer 1.The modality exception may reflect a closed class that classifiers can memorize; named entities being out of vocabulary may also matter.
- Architectural variants: Bidirectional encoders improve translation by +1-2 BLEU and SEM tagging by +3-4% accuracy, while residual connections yield up to +0.6% tagging gains.Both variants preserve the main pattern: POS benefits less from upper layers, whereas SEM improves with layer 4 representations.
5 Related Work
Prior work used neural MT hidden vectors to quantify linguistic information, mainly for syntax and morphology; this paper extends that approach to semantic tagging and compares it with POS tagging.
- Quantitative analysis: Neural-network analyses range from hidden-unit visualization to quantitative correlations between internal representations and linguistic properties.The cited work spans speech and language-processing models.
- Prior neural MT work: Shi et al. and Belinkov et al. used neural MT hidden vectors to predict linguistic properties, focusing on syntax and morphology rather than semantics.This work applies the same general methodology to semantic tagging and compares the results with POS tagging.
6 Conclusion
The paper studies how NMT representations encode linguistic information across layers and examines the effects of layer depth and target language on POS and semantic tagging. It identifies this analysis as a step toward understanding NMT representations and extending evaluation to relational tasks.
- Conclusion: The study examines linguistic information learned at different NMT layers, including the effects of layer depth and target language on POS and semantic tagging.The evaluation focuses on hidden representations in NMT models.
- Future work: Future work will extend the analysis to syntactic and semantic tasks that require building relations, such as dependency relations and predicate-argument structure.The authors connect this direction to understanding how semantic properties are learned in NMT.