Source-linked AI summary
Linguistic Input Features Improve Neural Machine Translation
Rico Sennrich, Barry Haddow
TL;DR
Neural MT performs strongly with little external linguistic information, leaving open whether explicit linguistic features are redundant. The paper extends the attentional encoder-decoder encoder to accept arbitrary feature combinations and tests them across several translation settings. Linguistic features improve translation quality over word-only baselines across perplexity, BLEU, and CHRF3, though their benefits may depend on the setting and future model capabilities.
Problem
The paper investigates whether explicit linguistic input features benefit neural MT or are redundant because neural models can learn from raw text.
Method
The encoder embedding layer is generalized to concatenate separate embeddings for arbitrary input features, including lemmas, POS tags, dependency labels, and morphology.
Results
Linguistic features improve the word-only baseline across perplexity, BLEU, and CHRF3; BLEU gains on newstest2016 are 1.5 for German→English, 0.6 for English→German, and 1.0 for English→Romanian.
Takeaways & Limitations
The empirical evidence suggests that linguistic input features benefit neural machine translation across the tested language pairs and metrics.
Takeaways & Limitations
The usefulness of the tested features may change as neural architectures improve and may be especially dependent on translation settings such as resource level or source-language inflection.
Abstract
from arXiv · showhide
Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic information. In this paper we show that the strong learning capability of neural MT models does not make linguistic features redundant; they can be easily incorporated to provide further improvements in performance. We generalize the embedding layer of the encoder in the attentional encoder--decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature. We add morphological features, part-of-speech tags, and syntactic dependency labels as input features to English<->German, and English->Romanian neural machine translation systems. In experiments on WMT16 training and test sets, we find that linguistic input features improve model quality according to three metrics: perplexity, BLEU and CHRF3. An open-source implementation of our neural MT system is available, as are sample files and configurations.
1 Introduction
The paper asks whether explicit linguistic information remains useful for neural machine translation despite strong learning from raw text. It motivates linguistic features as aids for sharing inflectional information and resolving lexical and syntactic ambiguities.
- Neural MT achieves strong results with little external linguistic information, raising whether explicit features are redundant.
- Lemmatisation can reduce data sparseness by sharing representations across inflectional variants.
- Part-of-speech and dependency annotations may help disambiguate source words and syntactic relations.
- A baseline mistranslates English adjective close as German verb schließen, illustrating the value of part-of-speech information for lexical disambiguation.The intended German translation is nah.
- For German-to-English translation, source syntactic annotation could help attention learn which words to translate first under German verb-second order.The example topicalizes gefährlich, placing die Route after the verb, while the baseline preserves this unusual English order.
2 Neural Machine Translation
The system uses an attentional encoder-decoder architecture and extends its encoder input representation from words to arbitrary feature combinations. Separate feature embeddings are concatenated while preserving the total embedding size and the rest of the model.
- The neural MT system is an attentional encoder-decoder network with recurrent neural networks.Its implementation follows a Bahdanau-style architecture and uses beam search for translation.
- A bidirectional encoder reads the source sequence and concatenates forward and backward hidden states into annotation vectors.
- The recurrent decoder predicts each target word using its hidden state, the previous target word, and an attention-derived context vector.Attention computes the context as a weighted sum of encoder annotations, with alignment weights learned jointly.
- The main innovation represents each encoder input as a combination of features rather than only a word representation.
- Separate embeddings for an arbitrary number of features are concatenated into a fixed-size input vector, leaving other model components unchanged.The total concatenated embedding size matches the original embedding size.
3 Linguistic Input Features
The paper incorporates linguistic annotations into subword-based neural MT inputs, focusing on features that support generalization, disambiguation, and word-structure modeling. Word-level annotations are copied across their subword units after BPE segmentation.
- The generalized encoder supports an arbitrary number of input features, including linguistic annotations.
- Lemmas explicitly share information among word forms with the same base form and may improve data efficiency for infrequent variants.The paper uses finite-state lemmatisers with broad coverage, including infrequent word forms.
- BPE provides open-vocabulary translation with a fixed symbol vocabulary, while subword tags encode each word’s segmentation structure.
- German morphological features encode word-type information such as case, number, gender, person, tense, and aspect.All morphological features for a word are concatenated into a feature value, with underspecification represented specially.
- Each word receives one POS tag and one dependency label, with the latter identifying its syntactic relation to the head or ROOT.
- After BPE segmentation, word-level feature values are copied to every subword unit belonging to the word.This annotation procedure is illustrated in Figure 1.
4 Evaluation
The evaluation tests linguistic input features across German↔English and English→Romanian neural MT, using perplexity, BLEU, and CHRF3 alongside translation examples and feature-isolation experiments. Linguistic features generally improve quality, though gains vary with direction, baseline strength, and synthetic-data noise.
- Experimental setup: The systems use BPE-segmented inputs with word, morphological, POS, and dependency features represented in a fixed-size embedding layer.Word-level feature values are copied to each subword unit; the total embedding size remains 500 to control parameter count.
- German↔English results: For German→English, linguistic features improve development perplexity from 47.3 to 46.2 and raise newstest2016 BLEU from 31.4 to 32.9 and CHRF3 from 58.0 to 58.5.These are comparisons against the word-only baseline.
- German↔English results: For English→German, linguistic features improve development perplexity from 54.9 to 52.9, BLEU from 27.8 to 28.4, and CHRF3 by 1.2 on newstest2016.The supplied passage reports the CHRF3 improvement but truncates its paired endpoint values.
- Feature analysis: In isolation, most individual features beat the baseline in perplexity and improve BLEU on at least one test set, but their gains are not fully cumulative because feature information overlaps.Subword tags help English→German slightly but not German→English; dependency labels and morphology can encode overlapping grammatical-case distinctions.
- Synthetic-data results: With synthetic training data, German→English improves in perplexity from 45.2 to 44.1, BLEU from 37.5 to 38.5, and CHRF3 from 62.2 to 62.8.For English→German, perplexity improves from 49.7 to 48.4, while BLEU and CHRF3 differences are small and non-significant.
- English→Romanian results: English→Romanian BLEU improves by 1.0 with linguistic features for both parallel-only systems, 23.8→24.8, and synthetic-data systems, 28.2→29.2.The best system is reported as competitive with the best WMT16 submission.
5 Related Work
The paper situates its source-side linguistic-feature approach within earlier uses of linguistic information in neural models and contrasts its flexible encoder integration with prior translation mechanisms.
- Source-side linguistic features have been used in neural language modeling and other neural tasks, including syntactic parsing.
- The paper remains a sequence-to-sequence model while using dependency annotations, unlike tree-to-sequence work focused on unlabelled source-tree structure.
- Phrase-based factored translation models face data sparsity or independence assumptions because they cannot easily generalize to new feature combinations.
- The approach differs from treating lemmas as a second source text because its encoder and attention layers are shared across features.
6 Conclusion
The paper finds that linguistic input features benefit neural machine translation and introduces a general encoder extension tested across several language directions. It also identifies future conditions that may change their usefulness.
- Linguistic input features improve neural machine translation according to the paper’s empirical evidence.
- The generalized attentional encoder-decoder incorporates arbitrary numbers of input features and is tested with lemmas, POS tags, dependency labels, and morphological features.
- 1.5 BLEU for German→English, 0.6 BLEU for English→German, and 1.0 BLEU for English→Romanian improve over the baseline on newstest2016.
- The features’ benefit may decrease as neural architectures improve, while their usefulness may vary with translation settings such as resource level and source-language inflection.