Source-linked AI summary
Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning
Jan Niehues, Eunah Cho
TL;DR
Neural machine translation has left linguistic annotations such as POS under-explored, despite their use in statistical machine translation. This paper jointly trains translation and linguistic tasks in an attention-based model, reporting gains in translation and POS tagging. The reported translation improvements reach 1.5 BLEU points and 2 CharacTER points.
Problem
Linguistic annotations such as POS have been used in statistical machine translation but remain under-explored in neural machine translation, whose models are often trained only on parallel data.
Source: §1 · p. 1Method
The paper jointly trains NLP tasks in an attention-based model to integrate linguistic resources directly into neural machine translation.
Source: §1 · p. 1Results
Up to 1.5 BLEU points and 2 CharacTER points are gained in translation, and POS tagging improves 30% to 50% relatively with multi-task learning.
Source: §6 · p. 8Takeaways & Limitations
The results suggest multi-task learning is promising for exploiting annotated resources, especially in low-resource conditions where annotation is time-consuming and expensive.
Source: §6 · p. 8The NMT system is sometimes unable to translate names or rare words correctly.
Source: §5.4 · p. 8
Abstract
from arXiv · showhide
Linguistic resources such as part-of-speech (POS) tags have been extensively used in statistical machine translation (SMT) frameworks and have yielded better performances. However, usage of such linguistic annotations in neural machine translation (NMT) systems has been left under-explored. In this work, we show that multi-task learning is a successful and a easy approach to introduce an additional knowledge into an end-to-end neural attentional model. By jointly training several natural language processing (NLP) tasks in one system, we are able to leverage common information and improve the performance of the individual task. We analyze the impact of three design decisions in multi-task learning: the tasks used in training, the training schedule, and the degree of parameter sharing across the tasks, which is defined by the network architecture. The experiments are conducted for an German to English translation task. As additional linguistic resources, we exploit POS information and named-entities (NE). Experiments show that the translation quality can be improved by up to 1.5 BLEU points under the low-resource condition. The performance of the POS tagger is also improved using the multi-task learning scheme.
1 Introduction
Neural machine translation often relies only on parallel data, leaving other linguistic knowledge sources underused. This work jointly trains NLP tasks in an attention-based model and examines how task choice, training schedule, and parameter sharing affect performance.
1 Introduction
- End-to-end neural models are often trained only on parallel data, whereas POS information has helped model word reordering in statistical machine translation.The paper notes that linguistic annotations have been used more extensively in SMT than in NMT.Source: §1 · p. 1
- The study uses a multi-task model to train several NLP tasks jointly and integrate linguistic resources directly into neural models.The approach aims to improve performance by using information shared across tasks.Source: §1 · p. 1
- The authors report improvements on German-to-English translation measured with BLEU, BEER, and CharacTER.They also investigate secondary-task choice, training schedule, and parameter sharing.Source: §1 · pp. 1–2
- The paper identifies three design decisions for multi-task learning: which tasks to include, how to schedule training, and how much to share across task architectures.It presents multi-task learning as a way to integrate additional linguistic resources directly into neural models.Source: §1 · pp. 1–2
2 Related Work
Prior work incorporated linguistic information into neural systems and explored multi-task learning, including for sequence-to-sequence models. The section also outlines recurrent encoder-decoder translation and attention, which lets the decoder use weighted information from across the source sentence.
2 Related Work
- Earlier work integrated POS information into language models and neural machine translation, while multi-task learning was extended from other neural tasks to sequence-to-sequence models.The cited examples include POS as an additional word factor in NMT (Sennrich and Haddow, 2016) and sequence-to-sequence multi-task learning.Source: §2 · p. 2
- Multilingual machine translation explored both separate encoders and decoders and shared encoder-decoder designs across language-pair tasks.The examples cited are Firat et al. (2016b) and Ha et al. (2016); Johnson et al. (2016).Source: §2 · p. 2
2.1 Attention Models
- The section contrasts attention-based NMT with a basic recurrent encoder-decoder that compresses the source sentence into a fixed-size context vector.The basic encoder processes source words sequentially and the decoder generates the target sequence one word at a time.Source: §2.1 · p. 2
- In an attention-based encoder-decoder, the decoder uses a weighted sum of encoder hidden states to focus on different source positions as it generates each target word.The bidirectional encoder provides information from the whole source sentence at every encoder position.Source: §2.1 · p. 2
3 Multi-task Learning
The paper casts translation, POS tagging, and named-entity tagging as tasks for an attention-based encoder-decoder, then varies which components they share and how examples are scheduled. It also uses known target lengths when decoding label sequences.
3 Multi-task Learning
- The tasks are German-to-English translation, POS tagging, and named-entity tagging, with the tagging tasks modeled as translating words into labels.The encoder-decoder framework is used across the tasks, which differ in their target sequences.Source: §3 · pp. 2–3
3.1 Architecture
- The baseline uses separate encoders, attention models, and decoders for each task, while the multi-task architectures vary the degree of parameter sharing.The paper notes that sharing may help transfer information, but may be harmful when tasks are very different.Source: §3.1 · p. 3
- The shared-encoder design uses one encoder for all tasks while keeping their attention models and decoders separate.It is the minimal degree of sharing considered in the experiments.Source: §3.1 · p. 3
- The shared-attention design adds a common attention model to the shared encoder, whereas the shared-decoder design shares task representations except for the final softmax layer.Figure 1 overviews the architectures and the layers shared in each.Source: §3.1 · p. 3, Figure 1 · p. 4
3.2 Training Schedule
- The default schedule shuffles single-task mini-batches from all tasks; the adapted schedule first trains on all tasks and then continues on the main task alone.The default schedule can concentrate training on the task with the most data when task datasets differ in size.Source: §3.2 · pp. 3–4
3.3 Target Length
- Translation target length is unknown, unlike the fixed label-sequence length, so decoding uses prior knowledge to generate label sequences of the correct length.The desired label length is not necessarily the same as the input length because inputs use subword units.Source: §3.3 · p. 4
4 Experimental Setup
The experiments combine German-to-English translation with German POS and named-entity tagging, using corpora of different sizes and domains. They evaluate translation with three metrics and tagging with error rates, using a shared attentional encoder-decoder setup.
4 Experimental Setup
- The experiments use German-to-English TED translation, German fine-grained POS tagging, and German named-entity tagging to study multi-task learning under sparse translation data.The translation setup uses TED data, while the linguistic resources come from other corpora.Source: §4 · p. 4
4.1 Data
- The translation training data comprises 4M WIT-corpus tokens, while the POS training data comprises 720K Tiger-corpus tokens from German newspaper text.The Tiger data is out of domain for the translation task; translation training examples are limited to sentences shorter than 60 words.Source: §4.1 · p. 4
- The German named-entity tagger is trained on 450K tokens from GermEval 2014 data extracted from Wikipedia.The training data comprises 24K sentences.Source: §4.1 · pp. 4–5
4.2 System Architecture
- All tasks and baselines use an attentional encoder-decoder with word embeddings, a bidirectional LSTM encoder, MLP attention, and conditional-GRU decoder units.The models are trained with Adam and early stopping based on validation log-likelihood.Source: §4.2 · p. 5
4.3 Evaluation
- Translation is evaluated with BLEU, BEER, and CharacTER, while POS tagging uses error rates on small and large label sets.The source and target corpora are jointly segmented into 40K byte-pair subwords after tokenization and true-casing.Source: §4.3 · p. 5, §4.1 · p. 5
5 Results
Multi-task training improves German-to-English translation and POS tagging in the reported low-resource experiments, while the results examine task combinations, training schedules, and parameter sharing. The examples also show improvements in translating compounds, repeated phrases, names, and rare words, though errors remain.
5.1 Initial experiments on the architecture
- Joint translation and POS training improved BLEU by more than 1 point, reduced characTER by more than 1.5 points, and improved BEER by more than half a point across all three architectures.The POS data were out-of-domain and about 20% the size of the parallel translation data.Source: §5.1 · p. 5
- The architecture sharing the most parameters beat baseline but underperformed the other two; separating tasks helped when they used different data, while sharing attention remained inconclusive.The authors focused later experiments on the shrd Enc and shrd Att architectures.Source: §5.1 · p. 5
5.2 Impact of design decisions
- The adapted shrd Enc model trained on all three tasks improved over baseline by 1.5 BLEU points, 1 BEER point, and 1.8 to 2 characTER points.Table 1 reports results for multi-task architectures on the machine translation task using BLEU, BEER, and characTER.Source: Table 1 · p. 6, §5.2 · p. 6
- Schedule effects varied by task and architecture: adaptation always helped the three-task system, had little effect for shrd Att, and had no effect for NE+MT.For the three-task combination, the adapted schedule improved average BLEU by 0.7, BEER by 0.5, and characTER by 1.2 points; for shrd Att, the default schedule won 10 of 18 cases.Source: §5.2 · p. 6
- Averaged across six systems, adapted schedules improved BLEU by 0.4, BEER by 0.2, and characTER by 0.4; they performed better in 25 of 36 paired comparisons.The measurements compare the same test set and metric under default and adapted schedules.Source: §5.2 · p. 6, Table 2 · p. 7
5.3 POS Tagging Performance
- POS adaptation improved every tested model, adding a third task helped in all cases, and architectures performed similarly; adapted three-task shrd Enc reached 5.06 small-tag-set error.The authors report that even the architecture sharing all components achieved comparable POS performance.Source: §5.3 · p. 7
- All multi-task models outperformed the POS baseline; the fine-grained tag-set error fell from 17.27 to 11.62, a reduction of more than 30%.The authors also report that the error rate was halved from the baseline performance of 10.13.Source: §5.3 · p. 7
5.4 Analysis and Examples
- Examples show improvements on German compounds and repeated phrases, and on copying Isaacson and translating Zerebralparese, though errors on names and rare words remained.The authors suggest that POS information may improve source-structure encoding; they also note cases where names or rare words were still mistranslated.Source: §5.4 · pp. 7–8
6 Conclusion
The authors conclude that jointly training attention-based encoder-decoder models on translation and other NLP tasks improves translation, with POS tagging also benefiting. Training schedule had the largest influence among the design choices studied, and encoder-only sharing achieved the best performance on both tasks.
6 Conclusion
- 1.5 BLEU points and 2 characTER points were the reported maximum translation improvements from multi-task learning.POS-tagging performance also improved by 30% to 50% relatively.Source: §6 · p. 8
- Training schedule had the largest influence on final performance; adapting the system to individual tasks made both POS and named-entity data beneficial for translation.The auxiliary corpora came from a different domain and were significantly smaller than the parallel data.Source: §6 · p. 8
- The best performance on both tasks was achieved with a model sharing only the encoder between tasks.Source: §6 · p. 9
- Future work will investigate adding target-language NLP tasks to the joint framework.Source: §6 · p. 9