Source-linked AI summary
Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English
Duygu Ataman, Matteo Negri, Marco Turchi, Marcello Federico
TL;DR
Fixed-size vocabularies constrain NMT translation when source words lack exact vocabulary matches, while rare and inflected forms contribute to data sparseness in morphologically rich languages. The paper uses unsupervised morphology learning to segment words into morphology-aware sub-word units while reducing the lexicon to a desired size. In Experiment 2.a, the method improves over the baseline at a vocabulary reduction rate of 4.25 (170K -> 40K).
Problem
Fixed-size vocabularies constrain NMT translation when source words lack exact vocabulary matches, while rare and inflected forms contribute to data sparseness in morphologically rich languages.
Source: §1 · p. 2Method
The paper uses unsupervised morphology learning to segment words into morphology-aware sub-word units while reducing the lexicon to a desired size.
Source: §1 · p. 3, §4.2 · p. 7Results
In Experiment 2.a, the method improves over the baseline at a vocabulary reduction rate of 4.25 (170K -> 40K).
Source: §6 · p. 9Takeaways & Limitations
The method achieves open-vocabulary translation while maintaining morphological information in sub-word units and can, in principle, be used with any language pair and direction.
Source: §1 · p. 3, §7 · p. 10
Abstract
from arXiv · showhide
The necessity of using a fixed-size word vocabulary in order to control the model complexity in state-of-the-art neural machine translation (NMT) systems is an important bottleneck on performance, especially for morphologically rich languages. Conventional methods that aim to overcome this problem by using sub-word or character-level representations solely rely on statistics and disregard the linguistic properties of words, which leads to interruptions in the word structure and causes semantic and syntactic losses. In this paper, we propose a new vocabulary reduction method for NMT, which can reduce the vocabulary of a given input corpus at any rate while also considering the morphological properties of the language. Our method is based on unsupervised morphology learning and can be, in principle, used for pre-processing any language pair. We also present an alternative word segmentation method based on supervised morphological analysis, which aids us in measuring the accuracy of our model. We evaluate our method in Turkish-to-English NMT task where the input language is morphologically rich and agglutinative. We analyze different representation methods in terms of translation accuracy as well as the semantic and syntactic properties of the generated output. Our method obtains a significant improvement of 2.3 BLEU points over the conventional vocabulary reduction technique, showing that it can provide better accuracy in open vocabulary translation of morphologically rich languages.
1. Introduction
Fixed-size vocabularies constrain NMT because source words must match vocabulary entries, while rare and inflected forms make this especially challenging for morphologically rich languages. The paper proposes morphology-aware vocabulary reduction and evaluates it for Turkish-to-English translation.
1. Introduction
- NMT’s fixed vocabulary limits translation of source words without exact vocabulary matches, while large vocabularies increase model complexity.These constraints make vocabulary size a trade-off between model complexity and coverage.Source: §1 · p. 2
- Turkish’s agglutinative suffixation creates many inflected forms, contributing to data sparseness in morphologically rich languages.Turkish inflections can express syntactic forms that English often expresses with separate words.Source: §1 · p. 2
- The proposed linguistically motivated method reduces vocabulary to a desired size while considering word morphology, and is evaluated on Turkish-to-English NMT.The paper compares it with statistical vocabulary reduction and supervised morphological segmentation.Source: §1 · p. 3
2. Neural Machine Translation
The paper’s NMT system uses an attention-based encoder-decoder with bidirectional encoding and autoregressive target-word prediction. Its recurrent components are GRUs, and vocabulary size affects model dimensions and computational cost.
2. Neural Machine Translation
- A bidirectional RNN encoder maps sparse one-hot source words to dense encoder hidden states.The model is based on the encoder-decoder and attention models described in Bahdanau et al. (2014).Source: §2 · p. 3
- A unidirectional decoder predicts each target word from the previous target word, decoder state, and an attention-weighted context vector.Attention weights combine encoder hidden states using the previous target word, previous decoder state, and corresponding encoder state.Source: §2 · p. 3
- Both RNNs use GRU gates, and embedding and hidden-layer dimensions scale with vocabulary size, increasing parameters and computation for larger vocabularies.Source: §2 · p. 3
3. Related Work
Prior vocabulary-reduction methods either rely on statistics without modeling morphology or incorporate morphology while limiting the extent of reduction. The paper identifies the absence of a method that combines morphology-aware units with reduction to any target vocabulary size.
3. Related Work
- BPE merges frequent character sequences but can split Turkish words inside roots or suffixes, producing semantic shifts and incorrect English inflections.The paper also notes that BPE dictionaries can contain repeated surface forms of the same lemma.Source: §3 · p. 4, Table 2 · p. 4, Table 3 · p. 4
- Character-level methods rely on statistics to capture morphological rules and are reported to generate spurious words.They use deep neural networks to compose character representations and new morphological forms.Source: §3 · pp. 4–5
- Morphological-analysis segmentation and Morfessor-based approaches add linguistic structure but have limitations in morphological description, vocabulary-size control, or sub-word ambiguity.The analyzer-based method does not fit any desired vocabulary size; the described Morfessor approach can set a target size but may produce semantically ambiguous units.Source: §3 · p. 5
- The stated gap is a method that considers individual sub-word morphology while reducing vocabulary to any given rate.Source: §3 · p. 5
4. Linguistically Motivated Vocabulary Reduction
The paper pairs supervised morphological segmentation, used to assess generalization, with an unsupervised method that learns morphology while targeting a vocabulary constraint. The unsupervised method modifies MAP optimization to favor a desired lexicon size.
4.1. Supervised Morphological Segmentation
- The supervised comparator separates Turkish roots and suffix tags into tokens, retaining morphological information while reducing sparsity from inflection and allomorphy.It uses morphological analysis and disambiguation, then adds an end-of-word symbol to each analyzed word.Source: §4.1 · pp. 5–6
4.2. Unsupervised morphological segmentation
- The unsupervised method extends Morfessor Flatcat, using category-based HMM segmentation and a flat lexicon so splits respect possible sub-word categories.It optimizes segmentation-model complexity under a vocabulary-size constraint.Source: §4.2 · p. 6
- The MAP model balances corpus likelihood and model complexity; its morpheme prior represents usage and form, with usage based on frequency, length, and left/right perplexities.Morpheme form refers to physical properties that distinguish morphemes in the lexicon.Source: §4.2 · p. 6
- A desired output lexicon size is used to set regularization from the initial and target vocabulary sizes, and as a convergence limit.This parameter controls vocabulary reduction and is also intended to minimize model convergence time.Source: §4.2 · p. 7
5. Experimental Set-up
The experiments compare segmentation methods in a controlled no-OOV setting and in two more realistic settings with different data distributions and vocabulary-reduction rates. The setup uses TED data for Experiments 1 and 2.a, and adds generic data for Experiment 2.b.
5. Experimental Set-up
- Experiment 1 compares supervised segmentation, the proposed method, BPE, and no segmentation in a controlled setting without test-set OOVs.The supervised method is used to assess whether segmentation captures morphology; both unsupervised methods are trained only on the source side for a fair comparison.Source: §5 · p. 7
- Experiments 2.a and 2.b test real-case vocabulary reduction, with generic data added in 2.b to increase data sparsity.Both use the same test set, and the source side is segmented with different methods while the target side uses BPE.Source: §5 · pp. 7–8
- TED Talks Turkish–English data is used in Experiments 1 and 2.a, while Experiment 2.b uses TED plus generic-domain data.Development and test sets contain 1,000 sentences each; the remaining data is used for training.Source: §5 · p. 8
- The NMT systems use Nematus with 1,024-dimensional hidden layers and embeddings, Adagrad, dropout, and 50 training epochs.The source dictionary is 40,000 in Experiment 1 and 30,000 in Experiment 2; the target dictionary is 40,000.Source: §5 · p. 8
- Modified Morfessor FlatCat models use lexicon sizes of 40,000 in Experiments 1 and 2.a and 30,000 in Experiment 2.b.The model adds output lexicon size as an input parameter; performance is evaluated with BLEU, TER, and CHRF3.Source: §5 · p. 8
6. Results and Discussion
The proposed method outperforms BPE in the controlled experiment and in both real-case experiments, including under increased data sparsity and stronger vocabulary reduction. Example translations illustrate how it preserves distinct meanings that BPE can conflate.
6. Results and Discussion
- In Experiment 1, the method outperforms BPE by 2.2 BLEU, 4.8 TER, and 1.6 CHR3F points.Its performance is slightly higher than the supervised method.Source: §6 · p. 9
- For ağ (“net”) and ağla (“to cry”), BPE assigns the same root, while the proposed method preserves the distinct meanings.The paper describes the BPE embedding as semantically ambiguous and its translations as unreliable in this example.Source: §6 · p. 9
- The method outperforms the baseline by 2.3 BLEU points; in the most challenging Experiment 2.b, with varying-domain training data and source vocabulary reduced from 270K to 30K, it gains 1.0 BLEU point.Source: §6 · p. 10
7. Conclusion
The paper concludes that its unsupervised, linguistically motivated vocabulary reduction method addresses vocabulary limits in NMT for morphologically rich languages. It maintains morphological information at the sub-word level and performs better than a statistical reduction method.
7. Conclusion
- The method is unsupervised and estimates a fixed-size dictionary of sub-word units using their individual morphological properties.It is proposed for open-vocabulary translation while maintaining a linguistic notion at the sub-word level.Source: §7 · p. 10
- The method addresses vocabulary limitation in NMT for morphologically rich languages.The paper frames this as an open issue in translation for these languages.Source: §7 · p. 10
- The evaluation reports significantly better performance than a statistical vocabulary-reduction method.The paper attributes the improvement to bringing a linguistic notion into segmentation.Source: §7 · p. 10