PaperlayerView exact revision on arXiv

Source-linked AI summary

What to Pre-Train on? Efficient Intermediate Task Selection

Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, Iryna Gurevych

arXiv:2104.08247v2cs.CL

TL;DR

Finding beneficial intermediate tasks is difficult because evaluating every dataset–model combination is infeasible, and adapter transfer can be harmful for some choices. The paper compares efficient selection methods across a broad task set and finds that embedding-based methods perform well, with the best methods achieving low average Regret@3.

  • Problem

    Identifying beneficial intermediate tasks is difficult because many candidate datasets and pretrained models make exhaustive evaluation infeasible, and prior selection methods lacked direct comparisons.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    The paper evaluates adapter-based sequential fine-tuning and consolidates task-selection methods that use metadata, task data, trained models, or combinations of these resources.

    Source: §1 · p. 2

  • Results

    Embedding-based methods perform better or often on par with more expensive approaches, and the best methods achieve average Regret@3 below 1%.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The evaluated methods support automatic identification of beneficial intermediate tasks, making exhaustive experimentation across task combinations unnecessary.

    Source: §8 · p. 9

    Proxy models are not studied on extractive QA tasks because those tasks cannot be directly transformed into classification tasks.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of candidate datasets as well as pre-trained language models, it has become infeasible to run the cross-product of all combinations to find the best transfer setting. In this work we first establish that similar sequential fine-tuning gains can be achieved in adapter settings, and subsequently consolidate previously proposed methods that efficiently identify beneficial tasks for intermediate transfer learning. We experiment with a diverse set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks. Our results show that efficient embedding based methods that rely solely on the respective datasets outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of less than 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training.

1 Introduction

Intermediate-task fine-tuning can improve target-task performance, but benefits vary and some task combinations hurt. This work establishes similar gains for adapters and develops efficient approaches to identify beneficial intermediate tasks.

1 Introduction

  • Intermediate-task fine-tuning can yield further gains, especially for low-resource targets, but many task combinations reduce performance.The abundance of labeled datasets and new pretrained language models motivates efficient selection of beneficial intermediate datasets.Source: §1 · p. 1
  • Adapter-based intermediate fine-tuning produces gains similar to full-model fine-tuning, but only a subset of intermediate adapters helps low-resource targets.Other intermediate adapters can considerably hurt performance, making efficient identification necessary.Source: §1 · p. 1
  • Adapters freeze pretrained transformer weights and train only newly introduced adapter weights, offering a modular and parameter-efficient alternative to full-model fine-tuning.The cited work reports on-par task performance and faster training for adapters.Source: §1 · p. 1
  • The work evaluates adapter-based sequential fine-tuning across 42 intermediate and 11 target tasks and identifies beneficial tasks without explicitly training on every candidate.It consolidates existing and new selection methods, analyzes data scenarios, task and model types, and estimates computational costs.Source: §1 · p. 2

2 Related Work

Prior work documents both gains and risks from sequential transfer, while methods for predicting beneficial sources span proxy estimators and dataset or task embeddings. The paper addresses a lack of direct comparisons and expands beyond BERT.

2.1 Transfer between tasks

  • Sequential intermediate-task training has improved many target tasks, but its success varies across intermediate tasks and can involve negative transfer.Phang et al. (2018) report gains, while Wang et al. (2019a), Yogatama et al. (2019), and Pruksachatkun et al. (2020) discuss forgetting and variable outcomes.Source: §2.1 · p. 2

2.2 Predicting Beneficial Transfer Sources

  • Transfer-selection research includes proxy estimators and dataset or task embeddings, but lacked direct method comparisons and had focused on BERT despite model-type differences.The paper aims to consolidate methods and include newer model types such as RoBERTa.Source: §2.2 · pp. 2–3

3 Adapter-Based Sequential Transfer

Adapter-based sequential transfer varies substantially across task combinations: positive gains are common, but losses are also frequent. The study evaluates 42 intermediate and 11 target tasks under a low-resource target setup.

3 Adapter-Based Sequential Transfer

  • The study establishes that adapter-based sequential transfer can produce gains, while around half of task combinations yield no positive gains.This motivates efficient identification of suitable intermediate tasks.Source: §3 · p. 3

3.1 Tasks

  • The study covers 53 English tasks: 42 intermediate and 11 target tasks, spanning classification, multiple choice, question answering, and sequence tagging.Tasks include QA from MultiQA, sequence tagging from Liu et al. (2019a), and classification tasks from (Super)GLUE.Source: §3.1 · p. 3

3.2 Experimental Setup

  • The experiments use BERT-base and RoBERTa-base with adapter-based two-stage fine-tuning, limiting target-task training to at most 1,000 examples.Adapters are trained on each intermediate task before fine-tuning on the target task; the target limit simulates a low-resource setting.Source: §3.2 · p. 3

3.3 Results

  • BERT and RoBERTa’s best-performing intermediate tasks overlap considerably; their transfer performances correlate at 0.94 overall and 0.68 when averaged per target task.Figure 1 compares the models’ transfer performance across target tasks after pretraining on the 42 intermediate tasks.Source: Figure 1 · p. 3, §3.3 · p. 3
  • Transfer gains and losses vary widely across combinations, with some performances up to 60% lower, potentially due to catastrophic forgetting.Figure 2 displays relative gains for intermediate-task choices across target tasks.Source: §3.3 · p. 3, Figure 2 · p. 4
  • 243 (53%) RoBERTa transfer combinations yield positive gains, while 203 (44%) yield losses; only five of 11 targets benefit on average.The mean transfer gain is 2.3%, but some combinations show losses up to 60%, illustrating the risk of selecting an unsuitable intermediate task.Source: §3.3 · p. 3, Table 1 · p. 4

4 Methods for the Efficient Selection of Intermediate Tasks

The paper groups task-selection methods by access to intermediate-task metadata, data, trained models, or both. The methods range from simple metadata rankings and embedding similarities to few-shot performance estimates and Fisher-based task embeddings.

4 Methods for the Efficient Selection of Intermediate Tasks

  • Selection methods are organized by whether intermediate-task data, trained models, or both are available, since access to both can be costly across large model repositories.The paper compares methods under these different availability assumptions.Source: §4 · p. 5

4.1 Metadata: Educated Guess

  • When neither intermediate-task data nor trained models are available, metadata-based selection can rank tasks by dataset size or select tasks with the same type.Size assumes more data implies better transfer; task-type selection assumes similar objectives transfer well and may be paired with random or size-based ranking.Source: §4.1 · p. 5

4.2 Intermediate Task Data

  • When intermediate-task data is available without trained models, TextEmb and SEmb rank tasks by cosine similarity between dataset embeddings and target-task embeddings.TextEmb averages final-layer language-model representations over examples and input tokens; SEmb uses sentence embeddings such as SBERT.Source: §4.2 · p. 5

4.3 Intermediate Model

  • With intermediate models available, few-shot fine-tuning ranks them by target-task performance after a few training steps, while proxy models estimate performance from model representations.The proxy approach uses kNN or linear/logistic regression on representations of target-task training inputs, then cross-validates to rank models.Source: §4.3 · p. 5

4.4 Intermediate Model and Task Data

  • When both task data and trained models are available, TaskEmb uses diagonal empirical Fisher Information Matrix entries as task embeddings, while FS-TaskEmb ranks models by embedding similarity before and after target-task training.FS-TaskEmb is intended to address the unreliability of few-shot performance scores with very few training instances.Source: §4.4 · pp. 5–6

5 Experimental Setup

The evaluation ranks intermediate adapters for each target task using transfer results from 462 experiments. It specifies method-specific training and scoring procedures for this selection task.

5 Experimental Setup

  • The selection methods rank intermediate adapters by expected performance on each target task, using transfer results from 462 experiments.The ranking objective applies to sequential adapter training.Source: §5 · p. 6

5.1 Hyperparameters

  • FSFT ranks intermediate adapters by target-task performance after one full epoch of fine-tuning, an optimistic estimate of few-shot transfer.The paper also investigates shorter runs of 5, 10, or 25 update steps in Appendix D.Source: §5.1 · p. 6
  • Proxy models use five-fold cross-validation; for sequence tagging, token instances are used and 1,000 target examples are randomly sampled to equalize training-set sizes.Proxy models are not evaluated on extractive QA because those tasks cannot be directly transformed into classification tasks.Source: §5.1 · p. 6
  • TaskEmb uses embeddings from fine-tuned, randomly initialized adapters, while FS-TaskEmb follows FSFT for one epoch or 50 update steps.SEmb uses Sentence-(Ro)BERT(a)-base models fine-tuned on NLI and STS tasks, matched to the target model type.Source: §5.1 · p. 6

5.2 Metrics

  • NDCG evaluates rankings using transfer results as relevances, while Regret@k measures the relative gap between the best of the top-k selections and the optimal task, normalized by optimal performance.Higher NDCG and lower regret indicate better selection.Source: §5.2 · p. 6

6 Experimental Results

Across the evaluated settings, simple domain-similarity indicators can select strong intermediate tasks for BERT and RoBERTa, often with zero Regret@3. The best selection approach depends on model family, while embedding-based methods offer lower-cost alternatives to few-shot fine-tuning.

6 Experimental Results

  • Random and Size baselines rank poorly across all intermediate tasks, while methods with access to only dataset or model information typically do better.Table 2 reports mean NDCG and Regret by target-task type for RoBERTa and BERT.Source: §6 · p. 6, Table 2 · p. 7
  • FSFT outperforms other approaches in most cases but requires downloading and fine-tuning every intermediate model, whereas TextEmb and SEmb are usually on par at lower cost.The lower-cost methods rely on access to intermediate-task vectors, such as those distributed with model repositories.Source: §6 · p. 7
  • TaskEmb performs well for BERT but does not transfer its results to RoBERTa, where it underperforms all methods and simple domain embeddings outperform it.For BERT, TaskEmb performs best or competitively across task types.Source: §6 · p. 7
  • TextEmb and SEmb are difficult to compare directly for the relevant classification tasks because the SBERT model was trained on NLI and STS-B, which also appear in the evaluated tasks.STS-B is included among the target tasks, while NLI is included among the intermediate tasks.Source: §6 · p. 7
  • Regret@3 is 0.0 in many cases, and the evaluated methods efficiently rank top intermediate tasks, reducing the number of necessary experiments.The authors conclude that simple domain-similarity indicators are suitable for selecting tasks for both BERT- and RoBERTa-based models.Source: §6 · p. 7

7 Analysis

Combining domain and task-type matching often gives the best overall results, while embedding-based methods offer a favorable efficiency/effectiveness trade-off. Transfer patterns and selector behavior also vary by task type and base model.

7 Analysis

  • Embedding-based cost estimates assume precomputed intermediate-task embeddings, while TaskEmb trains a target-task adapter for e epochs.The cost analysis estimates each transfer-source selection method's computational requirements.Source: §7 · p. 7
  • Combining domain and task-type match indicators often yields the best overall results, outperforming more computationally expensive methods.The authors report that this combination often produces the strongest overall results.Source: §7 · p. 7
  • Two orders of magnitude fewer computations were required by embedding-based methods than by fine-tuning approaches.The estimate uses 42 intermediate tasks and one target task with 1,000 examples each and average sequence length 128; the authors conclude SEmb has the most favorable efficiency/effectiveness trade-off.Source: §7 · pp. 7–8
  • SEmb variants based on BERT and RoBERTa models of Distill, Base, and Large sizes performed comparably on RoBERTa tasks.The authors describe SEmb as computationally efficient and invariant to model type in this comparison.Source: §7 · p. 8, Table 4 · p. 8
  • TaskEmb depends considerably on the base model: RoBERTa embeddings are more dispersed than BERT embeddings, which may hurt performance.The authors note that BERT TaskEmbs also appear to place related tasks near one another.Source: §7 · p. 8, Figure 3 · p. 8
  • Within-type transfer was consistently stronger, with the largest within/across-type difference for extractive QA; across-type transfer gains were negative for 8 of 11 target tasks on average.The authors suggest the relative homogeneity of QA intermediate tasks may partly explain the larger difference.Source: §7 · pp. 8–9, Figure 4 · p. 8
  • 77 of 99 cases showed improved NDCG when intermediate tasks of the target's type were preferred.Improvements were seen across target tasks and methods.Source: §7 · p. 9
  • Embedding-based approaches are sample-efficient, whereas few-shot fine-tuning approaches are not; rank fusion yields no consistent improvement over individual methods.The passage reports these as further analysis findings.Source: §7 · p. 9

8 Conclusion

The paper establishes that intermediate training can help adapter-based setups, but many transfer combinations reduce performance. It concludes that efficient embedding-based selection performs better or on par with more expensive approaches, with the best methods achieving average Regret@3 below 1%.

8 Conclusion

  • Around 44% of transfer combinations resulted in decreased performance in adapter-based setups.The paper also reports that intermediate pre-training can yield gains in these setups.Source: §8 · p. 9
  • Previously proposed best-performing approaches for BERT did not translate to RoBERTa.The authors report this difference when experimenting with different model types.Source: §8 · p. 9
  • Efficient embedding-based methods performed better or often on par with more expensive approaches, and the best methods achieved average Regret@3 below 1%.The authors conclude that these methods can identify beneficial intermediate tasks without exhaustive experimentation across task combinations.Source: §8 · p. 9
  • At broad scale, the authors say these methods can contribute to more sustainable and inclusive natural language processing.The claim is framed as a potential contribution when the methods are applied broadly.Source: §8 · p. 9

A Tasks

The experiments span a diverse set of intermediate and target tasks, with adapter training and evaluation settings specified for transfer comparisons. NDCG evaluates how well methods rank intermediate tasks by their observed transfer performance.

A Tasks

  • The benchmark contains 53 tasks across four task types, split into 42 intermediate tasks and 11 target tasks.The tasks include sequence classification/regression, multiple choice, extractive question answering, and sequence tagging.Source: A Tasks · p. 15

B Transfer training details

  • Adapters are trained for up to 15 epochs with early stopping, and target-task scores average five RoBERTa or three BERT restarts.The settings use a 10^-4 learning rate and batch sizes of 4–32; final scores come from test sets when publicly available, otherwise validation sets.Source: B Transfer training details · p. 16

C.1 NDCG

  • NDCG scores a predicted task ranking using each task’s mean target performance as relevance, then normalizes against the empirically perfect ranking.A score of 100% indicates a perfect ranking.Source: §C.1 · p. 16

C.2 Choice of metrics

The paper evaluates both full-ranking quality and the quality of top task selections. It favors metrics that reflect the greater importance of high-ranked tasks and differences in their actual transfer performance.

C.2 Choice of metrics

  • NDCG is preferred to rank correlation because it emphasizes correct rankings near the top, while correlation weights mismatches throughout the ranking equally.The authors report that Spearman correlation gave a poor indication of overall selection quality.Source: §C.2 · p. 16
  • Regret accounts for differences in target performance between top-ranked source tasks, unlike the average rank of the best-performing source task.The example contrasts methods that place the best source task at the same average rank but select alternatives with different performance.Source: §C.2 · p. 16
  • Figure 6 compares fine-tuning selection methods across checkpoints, with results averaged over RoBERTa targets.The caption identifies the figure’s scope but does not report a comparison outcome.Source: Figure 6 · p. 17

D Sample Efficiency

Embedding-based selection is tested with different amounts of target data, while few-shot fine-tuning is evaluated at different update steps. Rank fusion sometimes helps, but generally does not improve on its best component despite added computation.

D Sample Efficiency

  • With 10 target examples, SEmb reaches 73% NDCG, close to its 74.9% with 1000 examples, and consistently outperforms other embedding methods.Ranking quality can decrease substantially at the smallest sample size for other embedding methods.Source: D Sample Efficiency · p. 17
  • FS-TaskEmb ranks better at earlier checkpoints, but FSFT outperforms it later; fewer than 25 update steps do not reliably identify the best intermediate tasks.Both fine-tuning methods improve as the number of update steps increases.Source: D Sample Efficiency · p. 17

E Rank Fusion

  • Rank fusion improves on all included single methods in a few cases, but is usually no better than the strongest component and can substantially hurt selection.The authors note that fusion adds computational overhead without generally improving performance.Source: E Rank Fusion · p. 17

F SEmb Model Dependency

The SEmb model-variant experiments use RoBERTa transfer results and compare rankings across target-task types. The supplied table description identifies the variants and evaluation metrics, but does not provide comparative values.

F SEmb Model Dependency

  • The SEmb variant experiments use RoBERTa transfer results.The results are reported in Table 7.Source: F SEmb Model Dependency · p. 17
  • Table 7 distinguishes SEmb variants labeled D, B, and L, meaning Distill, Base, and Large.Source: Table 7 · p. 18
  • The table reports mean NDCG and Regret by target-task type, with higher NDCG and lower Regret indicating better rankings.The caption says the best overall scores are highlighted but does not state which variant wins.Source: Table 7 · p. 18

Related papers

Loading 2104.08247v2…