Source-linked AI summary
Multi-task Sequence to Sequence Learning
Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser
TL;DR
The paper addresses limited work combining multi-task learning with sequence-to-sequence models by proposing three complementary sharing settings. It finds that small auxiliary datasets can improve translation and parsing, while autoencoder and skip-thought objectives behave differently in this setting.
Problem
Little work had combined multi-task learning with sequence-to-sequence learning despite the popularity of both paradigms.
Method
The paper proposes one-to-many, many-to-one, and many-to-many multi-task sequence-to-sequence settings with shared encoders, decoders, or both.
Results
Multi-task training improved English–German translation by up to +1.5 BLEU points, achieved 93.0 F1 in constituent parsing, and showed different autoencoder and skip-thought effects.
Takeaways & Limitations
Small parsing and image-caption datasets can improve translation, while unsupervised objectives should be compatible with the supervised task and improve both intrinsic and extrinsic metrics.
Takeaways & Limitations
The unsupervised-task experiments used only 10% of the original WMT’15 monolingual corpora, leaving use of all monolingual data for future work.
Abstract
from arXiv · showhide
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.
1 INTRODUCTION
Multi-task learning had been widely studied, but little work combined it with sequence-to-sequence learning. This paper proposes three complementary sharing settings and reports improvements in translation, parsing, and unsupervised-learning comparisons.
- MTL aims to improve a task’s generalization performance using other related tasks.
- Seq2seq learning uses recurrent neural networks to map variable-length input sequences to variable-length output sequences.
- The paper proposes one-to-many, many-to-one, and many-to-many MTL settings for sharing encoders, decoders, or both across tasks.The tasks include machine translation, constituency parsing, image caption generation, and unsupervised objectives.
- +1.5 BLEU points over strong single-task baselines were obtained for English–German translation on WMT benchmarks.
- 93.0 F1 established a new state-of-the-art result in constituent parsing.
- Autoencoder helped less on perplexities but more on BLEU scores than skip-thought in the MTL setting.
2 SEQUENCE TO SEQUENCE LEARNING
Sequence-to-sequence models use an encoder–decoder framework to model conditional mappings between variable-length sequences. Their designs differ in recurrent architecture, recurrent-unit type, and whether the decoder accesses encoder hidden states through attention.
- Seq2seq learning directly models p(y|x), mapping an input sequence to an output sequence through an encoder–decoder framework.The encoder computes a representation s, and the decoder generates the output one unit at a time.
- Recurrent neural networks are the standard model for sequential data in recent seq2seq work.
- Seq2seq systems vary by architecture, including unidirectional, bidirectional, and deep multi-layer RNNs.
- They also vary by recurrent-unit type, particularly long-short term memory and gated recurrent units.
- Attention replaces a single encoder-state representation with access to the set of encoder hidden states for handling long input sequences.
3 MULTI-TASK SEQUENCE-TO-SEQUENCE LEARNING
The paper organizes multi-task sequence-to-sequence learning into one-to-many, many-to-one, and many-to-many architectures, then defines shared-task training with task-specific update proportions. It also investigates autoencoder and skip-thought objectives using monolingual data.
- Multi-task settings: The framework covers one-to-many, many-to-one, and many-to-many settings, selected according to whether encoders, decoders, or both can be shared.One-to-many uses one encoder and multiple decoders; many-to-one reverses this; many-to-many uses multiple encoders and decoders.
- 3.1 ONE-TO-MANY SETTING: In the one-to-many setting, a shared encoder feeds separate decoders for constituency parsing, German translation, autoencoding, or skip-thought prediction.The decoder outputs are respectively tags, German words, the same English sequence, or a related English sequence.
- 3.2 MANY-TO-ONE SETTING: In the many-to-one setting, multiple encoders share one decoder for tasks such as machine translation and image caption generation.This setting can also use large amounts of monolingual data on the target side.
- 3.3 MANY-TO-MANY SETTING: In the many-to-many setting, machine translation is combined with two unsupervised objectives over source and target languages using multiple encoders and decoders.The paper studies autoencoders on monolingual corpora and also investigates skip-thought vectors, adapting the latter by splitting unordered sentences into halves.
- 3.5 LEARNING: Training switches among tasks by sampling task i with probability αi, so αi controls its proportion of parameter updates.The first task is the reference task with α1 = 1.0 and N prescribed updates; other tasks receive αi ·N updates for comparison with single-task training.
- Parameter sharing: When an encoder or decoder is shared, both its recurrent connections and corresponding embeddings are shared.
4 EXPERIMENTS
The experiments evaluate multi-task sequence-to-sequence learning across translation, parsing, image captioning, and unsupervised objectives. Small amounts of auxiliary-task training improve translation and parsing, while autoencoder and skip-thought objectives show different metric behavior.
- Experimental scope: The evaluation covers constituency parsing, image caption generation, machine translation, and unsupervised learning tasks.
- Data and setup: The experiments use WMT’15 English⇆German translation data, shared 50K-word vocabularies, and newstest2013 for validation.
- Translation and parsing: +1.5 BLEU points over the single-task baseline is achieved by adding parsing with a 0.01 mixing ratio, while larger ratios overfit the small PTB corpus.The ratio corresponds to one parsing mini-batch per 100 translation mini-batches.
- Translation and parsing: +8.9 F1 points over the baseline is obtained for parsing in the small-corpus setup, although attention-free systems are not expected to reach the strongest parsing performance.The authors note that attention is crucial for strong performance on the small PTB corpus.
- Large parsing resources: +0.9 BLEU points is achieved when translation is jointly trained with parsing on the high-confidence corpus.The trend is consistent with the small-corpus parsing experiments.
- Parsing results: The multi-task parsing system reaches 93.0 F1 by ensembling six models trained with translation mixing ratios of 0.1, 0.05, and 0.01.At a 0.05 ratio, one multi-task system reaches 92.4 F1, on par with the best single system reported by Vinyals et al. (2015a).
- Unsupervised objectives: +0.5 BLEU scores are obtained when translation is coupled with autoencoding at a 0.05 mixing coefficient, but larger coefficients reduce translation performance.
5 CONCLUSION
The paper finds that multi-task learning improves attention-free sequence-to-sequence models across translation and parsing, while unsupervised objectives behave differently in the multi-task setting. It also identifies the lack of attention as a limitation and leaves multi-task learning with attention for future work.
- Multi-task learning improves the performance of the paper’s attention-free sequence-to-sequence model.
- Training with syntactic parsing and image caption data improves translation despite those datasets being much smaller than typical translation datasets.
- The paper establishes a new state-of-the-art constituent parsing result using an ensemble of multi-task models.
- Autoencoder and skip-thought objectives behave differently in the multi-task setting involving translation.
- The sequence-to-sequence models do not use attention, so multi-task learning with attention remains future work.