Source-linked AI summary
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
TL;DR
It is unclear when intermediate-task training improves target-task performance. This paper studies 110 RoBERTa intermediate-target pairs with 25 probing tasks and finds that complex reasoning and inference tasks tend to transfer best, while probing evidence remains diffuse and may be affected by pretraining knowledge forgetting.
Problem
Intermediate-task training does not always improve target-task performance, and the conditions under which it helps remain unclear.
Method
The paper conducts a large-scale RoBERTa study of 110 intermediate-target task pairs and evaluates trained models with 25 probes covering semantic and syntactic phenomena.
Results
Tasks requiring complex reasoning and inference, including Cosmos QA and HellaSwag, tend to work best as intermediate tasks, while higher-level semantic probes correlate more with target-task performance than lower-level skills.
Takeaways & Limitations
Specific skills driving positive transfer remain difficult to identify, supporting the need for broader and deeper probing tasks and more efficient transfer-learning mechanisms.
Takeaways & Limitations
Intermediate-task training may cause forgetting of knowledge acquired during pretraining, potentially limiting the interpretation of transfer results.
Abstract
from arXiv · showhide
While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate-target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level inference and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abilities such as coreference resolution. However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings.
1 Introduction
Intermediate-task training can improve pretrained models, but its benefits vary across target tasks and the conditions enabling transfer remain unclear. This study examines broad task combinations and probing skills to investigate when and why transfer works.
- Motivation: Intermediate-task training can improve pretrained models before target-task fine-tuning, but it does not always improve target-task performance.The approach is also known as STILTs.
- Study design: The study uses a broader and more diverse task set, additional probing tasks, and RoBERTa as its pretrained base model.
- Research questions: The study asks which intermediate tasks transfer broadly, which linguistic skills they teach, and which skills support success on particular target tasks.
- Research questions: The authors distinguish learning skills during intermediate-task training from demonstrating that those skills help performance on a target task.
- Study design: Experiments cover 11 intermediate tasks, 10 target tasks, and 25 probing tasks, producing 110 intermediate–target task pairs.
- Preview of findings: Natural language inference and commonsense-reasoning question-answering tasks are generally useful intermediate tasks, while input-noising skills correlate with target performance more than several low-level skills.
2 Methods
The methods construct a two-stage transfer pipeline in which RoBERTa is trained on one intermediate task and then separately fine-tuned on target and probing tasks. The task suite spans question answering, reasoning, entailment, syntax, semantics, and sentence-level linguistic behaviors.
- Pipeline: The experimental pipeline starts with pretrained RoBERTa, applies intermediate-task training, and then fine-tunes separate copies on target or probing tasks.
- Pipeline: Single intermediate-task training isolates the effects of skills learned from individual intermediate tasks rather than combining them through multitask training.
- Intermediate tasks: The intermediate-task set covers question answering, commonsense reasoning, and natural language inference, alongside syntactic tasks such as CCG supertagging.
- Intermediate tasks: QAMR and QA-SRL represent predicate–argument relationships through question-answer pairs, with QA-SRL focusing specifically on verbs.
- Intermediate tasks: HellaSwag tests choosing the most plausible story continuation, while CommonsenseQA tests a range of commonsense knowledge with multiple-choice questions.
- Target tasks: The target tasks include SuperGLUE-style question answering, entailment, word-sense disambiguation, and coreference resolution, with additional commonsense and multi-hop reasoning tasks.
- Probing tasks: The probing suite includes acceptability judgments, edge-probing tasks for syntactic and semantic span relations, and SentEval sentence-classification tasks.
3 Experiments
The experiments evaluate intermediate-task transfer across target and probing tasks using repeated RoBERTa training runs and baseline comparisons. Performance differences are reported as macro-averaged deltas between models with and without intermediate-task training.
- Training and evaluation: The study uses RoBERTaLarge and task-specific hyperparameter sweeps over four learning rates and two dropout rates.The selected configuration for each task is reused across runs.
- Training and evaluation: Each intermediate-task model is separately fine-tuned on 10 target tasks and 25 probing tasks, alongside baseline models without intermediate training.
- Training and evaluation: Three repetitions across 11 intermediate tasks and baselines produce 35×12×3 = 1260 observations.
- Transfer measurement: Figure 2 reports each target or probing task’s macro-average performance delta between the baseline and the model with intermediate-task training.A positive delta indicates successful transfer.
4 Results and Analysis
Intermediate tasks requiring complex reasoning generally produce positive transfer, especially for smaller target tasks, but effects vary substantially across probing skills and task pairs. Correlations suggest higher-level semantic abilities matter more than low-level syntactic skills, while catastrophic forgetting complicates interpretation.
- Target Task Performance: Intermediate tasks requiring complex reasoning and inference, including MNLI and commonsense QA, tend to produce positive target-task transfer.SocialIQA is a notable exception, while SocialIQA, CCG, and QQP produce negative transfer across target tasks and most probing tasks.
- Target Task Performance: Intermediate-task dataset size has no substantial, consistent impact on downstream target-task performance.The authors test varying data amounts for five intermediate tasks.
- Target Task Performance: Smaller target tasks such as RTE, BoolQ, MultiRC, WiC, and WSC benefit most from intermediate-task training.CommitmentBank shows no positive transfer because the baseline already reaches 100% accuracy.
- Probing Task Performance: Intermediate-task training yields little or no improvement on SentEval probing tasks, while acceptability judgments generally improve except AJ-CoLA.Low-level syntactic probing performance changes relatively uniformly across intermediate tasks.
- Variation across Intermediate Tasks: Performance varies across higher-level probing tasks: SocialIQA and QQP often hurt Edge-Probing results, whereas EP-SRL and EP-DPR improve across intermediate tasks.Intermediate tasks that improve the greatest number of probing tasks also tend to perform well as intermediate tasks.
- Correlation Between Probing and Target Task Performance: Acceptability judgments are generally uncorrelated with target performance, while input-noising tasks show stronger correlations with target tasks.Input-noising tasks resemble RoBERTa’s masked-language-modeling pretraining objective.
- Correlation Between Probing and Target Task Performance: Some strong correlations are driven mainly by variation in negative rather than positive transfer.This pattern suggests that intermediate-task training can cause models to forget some masked-language-modeling ability.
- Correlation Between Probing and Target Task Performance: Semantic probing abilities, including coreference, semantic proto-role labeling, and dependency labeling, show the highest correlations with target tasks.The target-task set is also oriented toward semantics and reasoning.
5 Related Work
Prior work studied transfer through sequential or multitask intermediate training, task-pair prediction, cross-lingual transfer, and probing pretrained representations. This paper extends that literature with a broader task-pair study and probing analysis using RoBERTa.
- Sequential and multitask intermediate training have both produced positive transfer to target tasks.
- Earlier cross-task studies identified language modeling and natural language inference as among the most beneficial intermediate tasks.
- Prior reading-comprehension transfer work found mostly positive transfer, with the largest gains from combining multiple question-answering datasets.
- This paper differs by studying a larger, more diverse set of single-intermediate-task transfers with RoBERTa and probing learned skills.
- Related prediction work used source, target, and task-pair features in multitask settings without self-supervised pretraining.
6 Conclusion and Future Work
The study finds that reasoning-intensive intermediate tasks tend to transfer best, while probing results provide only diffuse evidence about the specific skills driving transfer. The authors identify probing coverage and pretraining knowledge forgetting as important directions for future work.
- Cosmos QA and HellaSwag, which require complex reasoning and inference, tend to work best as intermediate tasks.
- Intermediate tasks that improve RoBERTa broadly across probing tasks show the most positive downstream transfer.
- Higher-level semantic probes correlate more with target-task performance, but the correlations are too diffuse to identify specific transfer-driving skills.
- Future work should improve the breadth and depth of probing tasks and develop more efficient transfer-learning mechanisms.
- High correlations with probes resembling masked language modeling suggest that forgetting pretrained knowledge may influence the transfer analysis.
A Correlation Between Probing and Target Task Performance
The appendix reports correlation matrices linking probing-task and target-task performance across intermediate-task training and random restarts. It presents both Spearman and Pearson analyses.
- Figure 4 reports Spearman correlations between probing-task and target-task performance across intermediate-task training and random restarts.
- Figure 5 reports Pearson correlations for the same probing-task and target-task performance comparisons.
- Together, the matrices compare rank-based and linear correlations across the same training conditions.
B Effect of Intermediate Task Size on Target Task Performance
The appendix examines how intermediate-task dataset size affects downstream target-task performance. It reports macro-averaged target metrics across randomly sampled dataset-size splits for five intermediate tasks.
- Figure 6 evaluates intermediate-task dataset size as a factor in downstream target-task performance.
- The analysis covers five intermediate tasks selected to vary in original dataset size and transfer-learning effectiveness.
- Each Figure 6 subfigure fine-tunes RoBERTa using a variety of randomly sampled dataset sizes.
- Reported outcomes are macro-averages of each target task’s performance metrics for every dataset-size split.