Source-linked AI summary
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, George Karypis
TL;DR
Prior work emphasizes both scaling multi-task training and choosing transferable tasks, but whether small related groups can match large-scale training remains an open question. This study compares pre-finetuning across task groups and finds that scale helps on average, while related groups can be a cheaper alternative for known targets.
Problem
Prior work separately examines scaling diverse task sets and selecting transferable tasks, leaving whether small related groups can match large-scale training unresolved.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
When targets are unknown, scaling is effective but expensive; for a specific known target set, training on fewer related tasks is a cheaper alternative.
Source: §1 · p. 2, §6 · p. 5The publicly released datasets may contain biases against protected groups, and the authors did not manually check them for offensive content.
Source: C Limitations and Future Work · p. 9
Abstract
from arXiv · showhide
Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.
1 Introduction
The introduction frames a tension between scaling multi-task training and selecting transferable tasks, then studies whether related task groups can rival large-scale training.
1 Introduction
- Prior work uses supervised data either to scale the number of tasks (Khashabi et al., 2020; Aghajanyan et al., 2021) or select target-related tasks (Pruksachatkun et al., 2020; Vu et al., 2020).Source: §1 · p. 1
- Task-transferability research finds that intermediate-task choice affects downstream fine-tuning, with high variance and challenging transfer prediction (Vu et al., 2020; Pruksachatkun et al., 2020).Source: §1 · p. 1
- The study asks whether pre-finetuning on a small group related to target tasks can match large-scale training, extending transferability experiments to task groups.Source: §1 · p. 1
- The experiments pre-finetune on combinations of task-format groups, then evaluate on target tasks belonging to each group to examine scale and task selection.Source: §1 · p. 1
- On average, large-scale multitask pre-finetuning improves downstream performance, while smaller related-task sets are comparable at lower computational cost.The introduction also notes that unrelated groupings can hurt performance.Source: §1 · p. 1
2 Multi-task Setup
The setup groups pre-finetuning and unseen target tasks by task format, then compares group combinations to study how task choice and scale relate to target performance.
2 Multi-task Setup
- The study groups 29 pre-finetuning tasks and 12 unseen targets into sequence labelling, extractive QA, and classification based on task format.The authors describe task-format grouping as an imperfect heuristic and focus on this intuitive output-space grouping.Source: §2 · p. 2
- Task relatedness is operationalized as belonging to the same task group, so the setup compares format-based groups rather than learned task-similarity embeddings.The alternative based on learned task embeddings is noted, but not pursued.Source: §2 · p. 2
- The research question is how pre-finetuning task choice and task count affect target performance, assessed across combinations of task groups and targets in each group.Source: §2 · p. 2
3 Experiments
The experiments use XLM-Roberta with a shared encoder and task-specific heads, and compare pre-finetuning task combinations under a common training and evaluation protocol.
3.1 Model Details
- Experiments use the pre-trained XLM-Roberta base model, with a shared encoder and task-specific linear heads during pre-finetuning and newly initialized heads for target tasks.Source: §3.1 · p. 2
3.2 Training Details
- Training batches mix examples from multiple tasks while preserving dataset distributions; all tasks use cross-entropy without per-task loss scaling.Preliminary experiments favored omitting the recommended loss scaling.Source: §3.2 · pp. 2–3
- All seven combinations of the three task groups are pre-finetuned with uniform hyperparameters, then target-task validation performance is averaged over five seeds with standard deviations.Pre-finetuning uses batch size 128 and early stopping based on validation loss; the direct-finetuning baseline uses only the pre-trained model.Source: §3.2 · p. 3
4 Results
The full 29-task setup has the best average performance across target tasks, while task-group matching makes smaller pre-finetuning setups competitive on related targets. Mismatched groups can hurt performance, and larger or related-task setups tend to reduce run-to-run variability.
A large-scale target task-agnostic Multi-task step improves downstream performance
- SL+QA+C has the best average performance across all target tasks and the best score on five individual tasks.The trend holds in a standard multi-task training regime without loss scaling or regularized finetuning.Source: §4 · p. 3, Table 1 · p. 4
Related tasks transfer better
- Only-SL and Only-QA are on par with SL+QA+C on unseen SL and QA tasks, while mismatched setups drop 9.6% on SL with Only-C and 20.3% on QA with Only-SL versus baseline.The reductions in computational cost mainly reflect using fewer pre-finetuning examples.Source: §4 · p. 3, Table 2 · p. 4
Tasks interact differently, so selecting an optimal subset is hard
- SL+C and QA+C perform below baseline on QA and SL targets, respectively, whereas SL+QA is competitive with SL+QA+C across unseen tasks.At larger scales, random task subsets have also been observed to outperform subsets chosen with simple heuristics.Source: §4 · pp. 3–4
Multi-task training reduces the variability of downstream task performance
- SL+QA+C has the lowest average run-to-run variation, and pairwise setups average lower variation than single-group setups.Single-group setups also have lower variability on unseen tasks from their own group than on other groups.Source: §4 · p. 4, Table 1 · p. 4
5 Related Work
Prior work studies both the benefits of large-scale multi-task training and how source-task choice affects transfer. This paper extends transferability analysis from individual tasks to task groups and relates it to large-scale multi-task learning.
Large-Scale Multi-task Learning
- Large-scale multi-task training has improved performance on unseen tasks, while this paper connects that line of work with task transferability.Prior studies report uniform improvements across unseen tasks and gains in zero-shot and few-shot settings.Source: §5 · p. 4
Exploring Relationships Between Tasks
- Empirical studies of intermediate-task selection have produced mixed results, and learned task embeddings have exposed relationships among tasks.Examples include clusters of semantic and syntactic tasks and improvements from jointly learning sequence-tagging tasks.Source: §5 · pp. 4–5
- Task embeddings can predict transferability, while prior work largely identifies a single suitable source task; this study extends the analysis to task groups.The concurrent Aribandi et al. (2021) study examined task-family transfer; this work uses a shared RoBERTa encoder and also compares task-group pairs and performance variance.Source: §5 · p. 5
6 Conclusion
The paper connects task transferability with large-scale multi-task training. It concludes that scaling suits unknown targets, while a smaller related-task set can be a cheaper alternative for a known target set.
6 Conclusion
- When target tasks are unknown, multi-task scaling offers an effective way to obtain good downstream performance; for a specific target set, related tasks offer an effective, cheaper alternative.The authors also report that task groups interact differently, making an optimum subset harder to select as the set grows.Source: §6 · p. 5
- Variability across random restarts decreases on related target tasks and as the size of the multi-task step increases.Source: §6 · p. 5
A Dataset Details
The appendix describes sequence-labelling, classification, and extractive question-answering datasets used for pre-finetuning and evaluation. It also explains that examples from selected datasets are pooled into heterogeneous batches.
Sequence Labelling (SL) Tasks
- The sequence-labelling data use CONLL format and cover 15 datasets with 225,433 training examples; unseen targets are evaluated with F1.The listed datasets include CONLL, Ontonotes, ANEM, GMB, SEC Filings, Malware TextDB, Few-NERD, and SNIPS.Source: A Dataset Details · p. 8
Classification (C) Tasks
- Pre-finetuning classification tasks come from GLUE, with four unseen targets from SuperGLUE; the pre-finetuning set has 943,984 training examples.CB is evaluated with F1 and the other three target tasks with accuracy.Source: A Dataset Details · p. 8
Question Answering (QA) Tasks
- The six pre-finetuning QA tasks come from MRQA, with 435,624 training examples; the evaluation uses F1 across QA targets.QED and SquadAdv use a 4:1 train-validation split, and XQuad uses English data with original splits.Source: A Dataset Details · p. 8
Heterogeneous Batches
- For each selected combination of task groups, examples are pooled across the corresponding datasets to create heterogeneous batches.The paper identifies this batching approach as the chief reason smaller subsets reduce computational cost.Source: A Dataset Details · p. 8
B Training Details
The experiments use XLM-Roberta with a shared encoder and separate task-specific heads. Classification, sequence labelling, and extractive question answering use different encoder outputs to make predictions.
B.1 Model Details
- XLM-Roberta uses the <s> representation for sequence classification, token representations for sequence labelling, and a span prediction for extractive question answering.Each pre-finetuning task has a separate task-specific head.Source: §B.1 · p. 8
- The model indexes task-specific heads using a list of pre-finetuning tasks and task indices supplied with each batch.The outputs are fed to linear classifiers.Source: §B.1 · p. 9
- For target-task evaluation, the encoder checkpoint is loaded and a new task-specific head is initialized.Source: §B.1 · p. 9
B.2 Pre-Finetuning and Target Task Finetuning Details
Pre-finetuning uses validation loss for early stopping and learning-rate selection. Target-task finetuning uses the same stopping patience and learning-rate range, with performance selected across five random restarts.
B.2 Pre-Finetuning and Target Task Finetuning Details
- Pre-finetuning stops when the average per-task validation loss fails to improve for 3 epochs.Validation loss is computed separately for each task at each epoch.Source: §B.2 · p. 9
- Pre-finetuning uses batch size 128 and selects a learning rate from 1e−3 to 1e−5 using validation data.The remaining Adam parameters use Huggingface Trainer defaults.Source: §B.2 · p. 9
- Target-task finetuning stops after average validation loss fails to improve for 3 consecutive epochs and sweeps learning rates from 1e−3 to 1e−5.Reported performance is the best selection across 5 random restarts.Source: §B.2 · p. 9
B.3 Computational Cost
The relative computational-cost improvement from selecting fewer tasks mainly reflects having fewer examples, with heterogeneous batches. Pre-finetuning epoch runtimes rise across the reported task-group runs.
B.3 Computational Cost
- Fewer examples mainly account for the relative cost improvement when selecting a smaller task subset, with heterogeneous batches used.Source: §B.3 · p. 9
- 1131 seconds per epoch for Only-SL, 1643 seconds per epoch for Only-C, and 2237 seconds per epoch for Only-QA.Source: §B.3 · p. 9
- 2661 seconds per epoch for SL+C, 3413 seconds per epoch for QA+C, and 3079 seconds per epoch for SL+QA.Source: §B.3 · p. 9
- SL+QA+C takes 4884 seconds per epoch.Source: §B.3 · p. 9
C Limitations and Future Work
The authors identify limits in their task-format grouping and unequal group sizes, and note risks from using public datasets without manually checking offensive content.
C Limitations and Future Work
- Task grouping is fuzzy, and the format-based heuristic may be harder to apply to non-standard tasks.An optimal grouping for each target set could be found, but selecting it would be a combinatorial task requiring significantly more compute.Source: C Limitations and Future Work · p. 9
- Task groups contain unequal numbers of examples, and controlling example counts would not account for relative task difficulty.Some chosen QA tasks seem more difficult for the model; controlling group size is left as future research.Source: C Limitations and Future Work · p. 9
- Public datasets may contain biases against protected groups, and the authors did not manually check them for offensive content.Source: C Limitations and Future Work · p. 9