Source-linked AI summary
Exploring and Predicting Transferability across NLP Tasks
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
TL;DR
The paper asks when intermediate fine-tuning on another task improves downstream NLP performance and studies transferability systematically across 33 tasks. It combines broad transfer experiments with task embeddings for source-task selection, finding especially strong benefits for low-data targets and identifying task, domain, data-size, and complexity factors as relevant.
Problem
The conditions for successful intermediate task transfer remain unclear, and arbitrary source-task choices can adversely affect downstream performance.
Method
The paper evaluates intermediate BERT fine-tuning across 33 NLP tasks and learns task embeddings from BERT-based task characteristics to select source tasks by cosine similarity.
Results
Transfer benefits are especially pronounced for low-data target tasks, while task and domain similarity, source data size, and task complexity all relate to transferability.
Takeaways & Limitations
Task embeddings can predict source tasks that will likely improve target-task performance, providing a practical alternative to enumerating every source-target combination.
Takeaways & Limitations
The task-embedding methods do not capture all factors influencing transferability, including unintuitive low-data transfer cases.
Abstract
from arXiv · showhide
Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve performance? In this paper, we conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even when the source task is small or differs substantially from the target task (e.g., part-of-speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size. Overall, our experiments reveal that factors such as source data size, task and domain similarity, and task complexity all play a role in determining transferability.
1 Introduction
The paper studies when intermediate fine-tuning transfers between NLP tasks and develops task embeddings to predict useful source tasks. Across 33 tasks, transfer is especially beneficial for low-data targets, while task and domain similarity help explain successful transfer.
- Motivation: Prior conditions for successful transfer were unclear, and arbitrary source-task selection could hurt downstream performance.The paper therefore investigates which source tasks transfer positively to particular targets.
- Study scope: The study examines transferability across 33 NLP datasets spanning text classification, question answering, and sequence labeling.It evaluates over 3,000 task and data-regime combinations, including within-class and across-class transfer.
- Findings: Transfer learning benefits are strongest when target-task training data is scarce, including cases where the source task is small or superficially different.The study reports positive transfer beyond settings favored by source-data-size heuristics.
- Findings: Task and domain similarity are crucial factors for successful transfer, particularly in data-constrained regimes.The analysis contrasts these factors with source data size as criteria for selecting source tasks.
- Task embeddings: The paper learns task embeddings that represent task characteristics and predicts transferable source tasks using cosine similarity.The pipeline computes embeddings from BERT-based task information and selects source tasks from a precomputed library.
- Resources: The authors publicly release pretrained models, task embeddings for the 33 studied tasks, and code for embedding new tasks and identifying likely positive-transfer sources.This release supports applying the source-selection approach to new tasks.
2 Exploring task transferability
The paper evaluates transferability across 33 NLP tasks, three task classes, and three data regimes using intermediate BERT fine-tuning. It finds that transfer is often beneficial, especially for data-constrained targets, while task and domain similarity influence which sources transfer successfully.
- Experimental setup: The study applies the STILTs pipeline to 33 tasks spanning text classification/regression, question answering, and sequence labeling.Each experiment fine-tunes pretrained BERT on a source task before fine-tuning on a target task, covering in-class and out-of-class transfer.
- Experimental setup: Experiments cover FULL →FULL, FULL →LIMITED, and LIMITED →LIMITED regimes, with LIMITED using 1K randomly selected training examples and 20 restarts.The regimes separate the effects of source and target data availability while averaging over instability from small-data fine-tuning.
- Transferability factors: Transfer gains can occur with small source datasets, and factors beyond source size—including task similarity, domain similarity, and source-task complexity—matter in low-data settings.For example, HotpotQA is the best source for WikiHop, POS-PTB for POS-EWT, and SQuAD is often strongest within QA, partly reflecting shared Wikipedia domains.
- In-class transfer: +47.0% average relative gain occurs in FULL →LIMITED, compared with +5.4% across all tasks in FULL →FULL.Most target tasks benefit from in-class transfer, and tasks with fewer training examples show the largest gains, including RTE (+17.0 accuracy points) and CQ (+14.9 F1).
- Out-of-class transfer: Out-of-class transfer benefits most tasks, although gains are smaller than for in-class transfer and are lower for sequence labeling than for classification or question answering.Some results are intuitive, such as SQuAD transferring to QNLI, while POS tagging produces large gains for DROP in limited target regimes.
- Transferability factors: Source dataset size is not always the best heuristic for data-constrained targets; similar tasks such as STS-B, QQP, and MRPC often transfer better.In FULL →LIMITED, STS-B is the best source for MRPC and QQP, while MRPC is best for STS-B despite having fewer examples than MNLI and SNLI.
3 Predicting task transferability
The paper predicts transferable source tasks using embeddings that separately capture text/domain properties and task-specific information, then evaluates their rankings against observed transfer gains. TASKEMB and combined embeddings generally outperform baselines, though important transfer patterns remain uncaptured.
- Task embedding methods: TEXTEMB averages BERT token representations across dataset inputs to encode text and domain properties without using training labels.The embedding pools final-layer token representations for each input and averages them over the dataset.
- Task embedding methods: TASKEMB uses Fisher information from BERT parameters after task fine-tuning to encode information about the knowledge and reasoning required by a task.The method computes an empirical Fisher over the training dataset and explores embeddings from multiple BERT components.
- Evaluation: The study asks whether task-embedding similarity can predict which source task maximizes transfer gain for a target task.It evaluates cosine-similarity rankings using the rank of the best source task and NDCG for the full ranking.
- Results: TEXTEMB performs better than DATASIZE on average, especially in limited-data regimes, but underperforms significantly on classification/regression tasks.The authors associate this weakness with the greater domain diversity of classification/regression datasets.
- Results: TASKEMB frequently outperforms other methods across problem classes, data regimes, and transfer scenarios, while combined TEXTEMB and TASKEMB provide further slight improvements.The ensemble combines complementary signals, but the small additional gains suggest TASKEMB already partially encodes domain similarity.
- Limitations: The embeddings do not capture all transferability factors, including STS-B’s transferability to CR and DROP’s benefit from sequence-labeling sources in low-data regimes.The authors identify more sophisticated task embeddings as a direction for future work.
4 Related Work
Prior NLP work studies transfer through multitask learning, intermediate fine-tuning, and task-selection methods based on dataset features or learning curves. Task embeddings provide a more principled alternative for representing task relationships across problem classes.
- Transferability between NLP tasks: Multitask and transfer-learning methods share knowledge across tasks, often improving over standard single-task learning through regularization and transfer.Intermediate fine-tuning has also been studied as a way to improve downstream performance.
- Task representations: Figure 3 contrasts TEXTEMB’s domain-oriented task space with TASKEMB’s stronger emphasis on task similarity.Penn Treebank sequence-labeling tasks cluster under TEXTEMB, while the two part-of-speech tagging tasks connect under TASKEMB despite different domains.
- Transferability between NLP tasks: Prior intermediate-fine-tuning work reports that transferring from data-rich source tasks boosts target task performance.Related work also suggests that tasks requiring high-level inference and reasoning can work well as intermediate tasks.
- Identifying beneficial task relationships: Earlier task-selection methods use dataset characteristics and learning curves, but manually designed features may be time-consuming and fail to generalize across problem classes.This motivates more principled task representations for predicting beneficial task relationships.
5 Conclusion
The paper studies transferability across 33 NLP tasks and develops task embeddings to predict useful source tasks. It finds that data size, task and domain similarity, and task complexity are important factors, especially when data is limited.
- Study scope: The study examines transferability between 33 NLP tasks spanning text classification/regression, question answering, and sequence labeling.Its experiments use pretrained BERT with intermediate source-task fine-tuning followed by target-task fine-tuning.
- Conclusions: Transfer benefits are more pronounced when target training data is limited, and task embeddings can predict source tasks likely to improve target performance.The analysis identifies data size, task and domain similarity, and task complexity as crucial factors in effective transfer.
A.1 Tasks & datasets
The study covers 33 datasets across text classification/regression, question answering, and sequence labeling, with dataset characteristics organized for transfer analysis.
- Task coverage: 33 datasets span text classification/regression, question answering, and sequence labeling.The datasets are grouped by task class and summarized by their characteristics and training-data sizes.
- Classification and regression: The classification/regression collection includes nine GLUE datasets plus SNLI and SciTail.Reported metrics include F1, Spearman correlation, and accuracy, depending on the task.
- Question answering: Eleven question-answering datasets include SQuAD-1, SQuAD-2, NewsQA, HotpotQA, BoolQ, DROP, WikiHop, DuoRC, ComplexQuestions, and ComQA.The QA collection includes datasets with varied context and question-construction settings.
- Sequence labeling: The sequence-labeling collection includes CCG supertagging, POS tagging, and syntactic constituency ancestor-tagging tasks.The listed POS datasets are Penn Treebank and Universal Dependencies English Web Treebank.
A.2 Fruchterman-Reingold force-directed placement algorithm
The Fruchterman-Reingold algorithm places tasks as interacting particles, using transferability to determine attractive forces and minimize the system’s energy.
- Force-directed placement: Tasks are modeled as atomic particles or celestial bodies exerting attractive forces on one another.The force-directed placement simulates a molecular or planetary system.
- Transferability forces: Transferability between tasks specifies the forces that position related tasks toward one another.The placement uses these forces to organize tasks spatially.
- Energy minimization: The algorithm minimizes the energy of the task system through force-directed placement.
B Full results for fine-tuning and transfer learning across tasks
The paper evaluates fine-tuning and intermediate-task transfer across 33 NLP tasks, comparing in-class and out-of-class sources under full and data-limited regimes.
- Fine-tuning setup: BERT is fine-tuned end-to-end with task-specific output layers, while the feature extractor is pretrained.Whole-model fine-tuning performs better in most cases, except WNLI and DROP, possibly because of their adversarial nature.
- Data regimes: 1K randomly selected training examples define the LIMITED regime, with 20 random restarts used to report means and standard deviations.The FULL regime uses all available training data for the associated task.
- Transfer experiments: 3,267 source-target and data-regime combinations cover every possible task pair within and across the three problem classes.Transfer follows intermediate fine-tuning on the source task before fine-tuning on the target task.
- In-class results: In-class transfer tables use source tasks as rows and target tasks as columns, with diagonal cells representing target-only BERT fine-tuning.Positive transfers are blue, and the best results are highlighted in bold; LIMITED results report means and standard deviations across 20 restarts.
- Out-of-class results: Out-of-class transfer tables use a Baseline row for target-only BERT fine-tuning before comparing source-task transfers.