Source-linked AI summary
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, Donald Metzler
TL;DR
The paper addresses limited systematic evidence on scaling the number of tasks in NLP multi-task pre-training. It introduces EXMIX and ExT5, combining supervised multi-task training with self-supervised span denoising, and reports stronger benchmark performance and sample efficiency than T5 baselines. The study also finds that manually curating an ideal task subset is not straightforward, whereas broader task inclusion generally improves downstream performance.
Problem
Few studies have systematically examined how scaling the number of tasks affects NLP multi-task pre-training, while negative transfer complicates task selection.
Method
The paper introduces EXMIX, a collection of 107 supervised NLP tasks, and pre-trains ExT5 on supervised EXMIX combined with self-supervised C4 span denoising.
Results
ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA, and several tasks outside EXMIX, while improving sample efficiency.
Takeaways & Limitations
Broad and diverse multi-task pre-training generally improves downstream performance without requiring an expensive search for the best task subset.
Takeaways & Limitations
Task-family conclusions depend on representative datasets whose domains and nuances make them imperfect proxies for larger task families.
Abstract
from arXiv · showhide
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.
1 INTRODUCTION
The paper studies extreme scaling of supervised tasks for multi-task pre-training, introducing EXMIX and ExT5. It finds that manually selecting an ideal mixture is difficult, while broad supervised co-training improves transfer and sample efficiency.
- Multi-task pre-training remains insufficiently studied at large task scales, despite transfer learning’s importance for NLP.
- Extreme Multi-task Scaling doubles the task count of the largest prior study and focuses on multi-task pre-training rather than fine-tuning.
- ExT5 combines supervised EXMIX with self-supervised C4 span denoising and outperforms strong T5 baselines across several benchmarks and tasks outside EXMIX.
- Including EXMIX can reduce the number of pre-training steps needed for strong performance, improving sample efficiency.
- EXMIX contains 107 supervised NLP tasks and approximately 18M labeled examples across diverse task families.
- Fine-tuning transfer relationships do not straightforwardly identify an ideal pre-training mixture, making exhaustive task-subset search challenging and expensive.
2 THE EXMIX TASK COLLECTION
EXMIX is a unified, large-scale collection for studying multi-task transfer across diverse NLP task families. Experiments show that cross-family transfer is often negative, yet increasing task diversity generally benefits multi-task pre-training and improves sample efficiency.
- The EXMIX task collection: EXMIX contains 107 diverse English NLP tasks totaling 18M examples, formatted as text-to-text examples for shared multi-task training.The unified format avoids task-specific heads and losses.
- The EXMIX task collection: The collection covers classification, natural language inference, reading comprehension, closed-book question answering, commonsense reasoning, semantic parsing, dialogue, and summarization.
- Transfer relations between EXMIX tasks: Co-training across task families often causes negative transfer: 21 of 56 relationships worsen performance under a fixed data budget, rising to 38 of 56 under fixed compute.NLI helps most other families, while summarization often hurts and closed-book QA is highly sensitive.
- Transfer relations between EXMIX tasks: Within-family dataset performance is usually positively correlated, but exceptions occur, including an inverse relationship between GEM schema-guided dialogue and KILT Wizard-of-Wikipedia.
- Task scaling: With large batches, increasing the number of mixed tasks generally improves downstream performance, while small batches show less upward trend.The authors attribute the small-batch limitation partly to noisy multi-task gradients.
- Improving sample efficiency with EXMIX: EXMIX pre-training is more sample-efficient than vanilla self-supervised pre-training, reaching 75.8 SuperGLUE at 20k steps and exceeding fully pretrained BERT large by about 4%.
3 THE EXT5 MODEL
ExT5 is a T5-based encoder-decoder model pre-trained with supervised ExMix and self-supervised C4 span denoising. It consistently outperforms strong T5 baselines across in-mixture benchmarks and improves performance on several tasks outside ExMix.
- 3 THE EXT5 MODEL: ExT5 revisits multi-task pre-training to address negative transfer and catastrophic forgetting encountered in large-scale co-training.It is a sequence-to-sequence Transformer encoder-decoder model based on T5.
- 3.1 TRAINING EXT5: ExT5 combines C4 span denoising with supervised ExMix examples, controlled by the sampling ratio R between C4 and ExMix.The span-denoising objective matches the one used by T5.
- 3.1 TRAINING EXT5: ExT5 uses 1M pre-training steps, batch size 2048, sequence length 512, and approximately 1T total tokens, matching T5’s training steps and token count.The models use the T5.1.1 architecture and standard sequence-to-sequence cross-entropy loss without label smoothing.
- 3.2 RESULTS: ExT5 consistently outperforms strong T5 baselines across SuperGLUE, GEM, Rainbow, MSMarco, and closed-book question-answering datasets.The comparisons cover a range of model sizes.
- 3.2 RESULTS: +5%, 2.3%, +0.7%, and +0.2% are ExT5’s SuperGLUE gains for BASE, LARGE, XL, and XXL, respectively.On GEM, ExT5 outperforms T5 on 6 of 9 tasks; Rainbow improves by +0.7% over the authors’ T5 run on average.
- 3.2 RESULTS: ExT5 outperforms T5 on out-of-mixture machine translation, scientific reasoning, and named entity recognition tasks.The largest improvement occurs on ARC, while the trend also holds for NER and MT tasks without similar datasets in ExMix.
- 3.2 RESULTS: ExT5’s improved generalization can avoid the expense of pre-training again with ExMix plus each new target task.The authors suggest repeating pre-training only when the training-dataset collection grows substantially.
4 RELATED WORK
Related work covers multi-task learning, task-relationship analysis, task-selection methods, and self-supervised pre-trained Transformers. ExT5 is positioned as the first massive-scale multi-task pre-trained model and as scaling over diverse labeled tasks.
- Improving NLP models with Multi-task Learning: Earlier multi-task studies progressed from simple NLP tasks to setups using around 30 or 50 tasks and models up to 440M parameters.The cited work includes MT-DNN and a later study using approximately 50 tasks.
- Relationships amongst different tasks: Prior research analyzes relationships among traditional NLP tasks, classification and regression tasks, question answering, sequence labeling, and jointly trained QA and summarization tasks.Other work learns task relationships through optimization objectives or task embeddings.
- Choosing which tasks to transfer from: Task-selection research uses Bayesian optimization, multi-armed bandits, and Gaussian processes to identify relevant tasks or control mixing rates.These approaches address the choice of tasks or transfer languages for pre-training and transfer learning.
- Pre-trained Transformers: Transformer models such as T5, BERT, and GPT-3 rely on large unlabeled corpora for self-supervised learning, while supplementary pre-training tasks can improve performance.The paper presents its contribution as the first massive-scale multi-task pre-trained model.
- Scaling Laws: This work scales pre-training by increasing the number of high-quality, diverse labeled tasks rather than only model size or dataset size within one corpus.It frames task-count scaling as distinct from conventional Transformer scaling laws.
5 EPILOGUE
The paper acknowledges limits in representing task families and proposes multilingual ExT5 as future work. Its conclusion is that massive supervised multi-task pre-training strengthens downstream performance and sample efficiency.
- Limitations: Task-family analyses depend on the nature, domain, and expressiveness of representative datasets, which are only proxies for broader task families.The authors also caution that lexical metrics such as BLEU and ROUGE do not fully capture text-generation performance.
- Future Work: A multilingual ExT5 would need to balance both task families and task languages.The authors propose multilingual ExMix as a stronger basis for analyzing cross-language transfer.
- Conclusion: Negative transfer is common in fine-tuning, but scaling task count in multi-task pre-training enables strong downstream performance with better sample efficiency.The conclusion identifies ExMix and ExT5 as the paper’s main contributions.
ETHICS STATEMENT
The ethics statement identifies bias and environmental impact as important considerations for large pre-trained models. It notes that supervised pre-training may warrant investigation for bias mitigation and that sample-efficient training may reduce compute in future experiments.
- ETHICS STATEMENT: Large language models can capture biases from their pre-training data, but analyzing whether supervised pre-training mitigates bias is outside this paper’s scope.The authors suggest diverse values-targeted data as an alternative direction.
- ETHICS STATEMENT: ExMix improves sample-efficient pre-training relative to standard self-supervision, which the authors hope will save compute in future experiments.The statement also highlights the energy consumption and environmental impact of training large models.
REPRODUCABILITY STATEMENT
The ExT5 modeling and training code, along with its variants, is open-sourced through the Mesh Tensorflow and T5 libraries. ExMix uses publicly available datasets.
- ExT5 and its variants use open-sourced modeling and training code.The code is distributed through the Mesh Tensorflow and T5 libraries.
- ExMix is composed of publicly available datasets.
- The reproducibility resources cover both implementation code and training data sources.
A DATASETS
ExMix contains 107 supervised datasets assembled from diverse benchmark collections and task families. The listed collections span generation, dialogue, social-media analysis, language understanding, knowledge-intensive tasks, and commonsense reasoning.
- ExMix contains 107 training datasets, including grouped benchmark collections.The table identifies the datasets used from each collection, focusing on collections with English training data.
- GLUE and SuperGLUE: GLUE and SuperGLUE contribute broad language-understanding benchmarks, including entailment, similarity, classification, reasoning, and coreference tasks.
- KILT and Rainbow: KILT and Rainbow add knowledge-intensive retrieval and question-answering tasks alongside commonsense and social reasoning benchmarks.
- GEM: The GEM collection contributes English generation datasets such as Wiki-Lingua, CommonGEN, E2E, DART, ToTTo, and Wiki-Auto.
- DialoGLUE and TweetEval: DialoGLUE and TweetEval cover dialogue understanding plus emotion, emoji, irony, hate-speech, offensive-language, sentiment, and stance tasks.
B EXPERIMENTAL DETAILS
The experiments use Mesh Tensorflow with the T5 library. This section introduces the experimental details and implementation framework.
- The section reports experimental details for the models and training setup.
- The implementation framework connects the reported experiments to established Mesh Tensorflow and T5 codebases.
- The models were trained using Mesh Tensorflow and the T5 library.
B.2 DATASET EXPERIMENTAL SETUP
The paper uses task-specific fine-tuning and multi-task evaluation setups across SuperGLUE, GEM, CBQA, Rainbow, translation, ARC, and NER. Training configurations vary by task, including learning-rate sweeps, decoding strategies, and evaluation metrics.
- SuperGLUE: SuperGLUE is fine-tuned as a proportionately sampled mixture for 200k steps with batch size 128.Checkpoint selection follows the T5 convention, choosing the best checkpoint per task for comparison with individually fine-tuned models.
- GEM: GEM evaluation uses metric-based checkpoint selection and greedy decoding, with learning rates of 10^-3, 5 × 10^-4, and 10^-4.Checkpoint selection averages BLEU, ROUGE-1, ROUGE-2, and ROUGE-L on validation data.
- CBQA and Rainbow: CBQA and Rainbow use validation-based evaluation with learning-rate sweeps over 10^-3 and 10^-4.Rainbow co-trains on all datasets.
- WMT Machine Translation: WMT translation covers EnDe, EnFr, and EnRo with learning rate 10^-3, dropout 0.1, batch size 4096, and beam size 4.Models train for up to 400k steps and report peak validation BLEU with length penalty 0.6.
- ARC: ARC uses Challenge-set evaluation, batch size 32, and a learning-rate sweep over 10^-3 and 10^-4.
- CoNLL-03 NER: CoNLL-03 NER is converted to sequence-to-sequence prediction of ordered tags and entities, with sentence-level exact-match accuracy.A sentence is correct only when the predicted tag-and-entity sequence exactly matches the reference.
C DETAILED EXPERIMENTAL RESULTS
This section reports detailed SuperGLUE evaluation, including average model scores and full results across datasets and experimental subsections.
- SuperGLUE experiments evaluate models using an average score, with full results reported for all datasets.
- Table 12 contains full SuperGLUE results from Section 2.2.
- Table 13 contains full SuperGLUE results from Section 2.3.
- Table 14 contains full SuperGLUE results from Section 2.4.
- Tables 15 and 16 contain full SuperGLUE results from Sections 2.5 and 2.6, respectively.