PaperlayerView exact revision on arXiv

Source-linked AI summary

ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts

Akari Asai, Mohammadreza Salehi, Matthew E. Peters, Hannaneh Hajishirzi

arXiv:2205.11961v2cs.CL

TL;DR

Parameter-efficient tuning can trade performance for efficiency and often does not benefit from collections of other NLP tasks. ATTEMPT mixes reusable source prompts with a target prompt through attention; across 21 datasets, it outperforms prior prompt tuning and matches or outperforms stronger-parameter approaches.

  • Problem

    Parameter-efficient tuning can reduce task performance and often uses only target-task data rather than knowledge from large collections of other NLP tasks.

    Source: §1 · p. 1

  • Method

    ATTEMPT combines source-task soft prompts and a newly initialized target prompt through shared attention to form instance-wise prompts while keeping the original LM frozen.

    Source: §1 · p. 1

  • Results

    Across 21 diverse datasets, ATTEMPT significantly outperforms prior prompt-tuning approaches and matches or outperforms fully fine-tuned or parameter-efficient methods, especially on smaller datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pre-trained soft prompts support modular transfer that can flexibly combine, reuse, remove, or add tasks, while attention distributions expose task similarities.

    Source: §1 · p. 2, §7 · p. 9

    The evaluation covers classification and QA rather than long-sequence generation, and does not test non-English tasks.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

This work introduces a new multi-task, parameter-efficient language model (LM) tuning method that learns to transfer knowledge across different tasks via a mixture of soft prompts-small prefix embedding vectors pre-trained for different tasks. Our method, called ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task. During training, only the target task prompt and the attention weights, which are shared between tasks in multi-task training, are updated, while the original LM and source prompts are intact. ATTEMPT is highly parameter-efficient (e.g., updates 2,300 times fewer parameters than full fine-tuning) while achieving high task performance using knowledge from high-resource tasks. Moreover, it is modular using pre-trained soft prompts, and can flexibly add or remove source prompts for effective knowledge transfer. Our experimental results across 21 diverse NLP datasets show that ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully fine-tuned or other parameter-efficient tuning approaches that use over ten times more parameters. Finally, ATTEMPT outperforms previous work in few-shot learning settings.

1 Introduction

ATTEMPT combines reusable source-task soft prompts with a target prompt through shared attention, aiming to improve task performance without full-model tuning. The introduction reports gains across diverse datasets alongside parameter efficiency and modular transfer.

1 Introduction

  • ATTEMPT learns instance-wise prompts by attention-weighting reusable source-task prompts together with a newly initialized target prompt, while keeping the original LM frozen.The attention module is lightweight and can be shared and trained simultaneously across tasks.Source: §1 · p. 1
  • 0.4% of parameters are updated while ATTEMPT achieves competitive performance.The introduction contrasts this with full fine-tuning.Source: §1 · p. 1
  • Pre-trained prompts make multi-task transfer modular: source prompts can be combined, reused, removed, or supplemented, while learned attention distributions expose task similarities.A single LM with multiple pre-loaded soft prompts can perform multiple tasks.Source: §1 · p. 2
  • Across 21 datasets spanning diverse tasks, domains, and output formats, ATTEMPT significantly outperforms previous prompt-tuning approaches and matches state-of-the-art parameter-efficient methods or fully fine-tuned models, especially on smaller datasets.ATTEMPT is also effective in few-shot domain adaptation with 4–32 shots.Source: §1 · p. 2
  • Ablations suggest that learned attention, multi-task learning, and transfer from multiple tasks contribute to performance improvements.Attention distributions also show similarities between seemingly different tasks, such as entailment and paraphrase detection.Source: §1 · p. 2

2 Background and Problem Setup

The section frames target-task learning as a choice between full fine-tuning, parameter-efficient tuning, and multi-task transfer. ATTEMPT’s setup combines efficiency with transfer despite uncertainty about which source tasks will help.

2 Background and Problem Setup

  • Full fine-tuning specializes a pre-trained LM by optimizing all its parameters on target-task training data..Source: §2 · p. 2
  • Parameter-efficient approaches update fewer parameters, but can lose performance relative to fine-tuning or be sensitive to initialization.Prompt transfer can improve performance, while requiring substantial search.Source: §2 · p. 2
  • Massive multi-task pretraining can lack the flexibility to add or remove source tasks, including when tasks interfere.The section contrasts this with the goal of flexible task transfer.Source: §2 · p. 2
  • The target-task goal is to update a small parameter set while transferring knowledge from source tasks, without knowing in advance which tasks provide useful inductive bias.Seemingly different tasks may benefit from one another.Source: §2 · pp. 2–3

3 Method

ATTEMPT transfers source-task knowledge through prompts, combining them with a target-task prompt according to input-specific attention while keeping the language model frozen. Sharing attention across target tasks supports modular multi-task training and efficient inference.

3 Method

  • ATTEMPT forms an instance-specific prompt by attending over pre-trained source prompts and a new target prompt, then prepends it to the input of a frozen language model.Training updates only the target prompt and attention module; the source prompts and language model remain unchanged.Source: §3 · p. 3

3.1 Source Prompt Pre-training

  • Each source prompt is trained once by prompt tuning on a high-resource task, encoding task knowledge in soft embeddings that can transfer to multiple target tasks.A prompt is a sequence of m embeddings in the language model’s d-dimensional space; only those embeddings are updated during source-task training.Source: §3.1 · p. 3

3.2 Target Prompt Training

  • For a target task, ATTEMPT learns instance-wise prompts by interpolating its new task prompt with source prompts using attention scores.The target prompt and attention module are updated through the instance-wise prompt, while source prompts and the original language model stay fixed.Source: §3.2 · p. 3

3.2.1 Input-prompt Attentions

  • The attention module pools the input and prompt embeddings, projects the input into prompt spaces, and softmaxes their scores to obtain prompt weights.The authors use Layer Norm because without it the projected representation often grows quickly and gradients explode; a softmax temperature scales logits to avoid over-confident attention.Source: §3.2.1 · p. 4

3.2.2 Prompt Interpolation

  • The instance prompt is a weighted sum of prompts, with the target prompt always retaining influence and source prompts down-played when none is useful.Max-pooling adds no parameters and empirically performs slightly better than other pooling approaches.Source: §3.2.2 · p. 4

3.3 Multi-task Training and Inference

  • Multi-task training shares the attention module across target tasks while retrieving each task’s prompt by its task ID; shared attention updates at every iteration.The authors say this loosely connects tasks, can help when target-task data is small, and reduces parameters per task and inference-time overhead. At inference, one model load can serve multiple target tasks with their prompts and shared attention.Source: §3.3 · p. 4

3.4 Parameter Efficiency of ATTEMPT

  • With T5-XL as the backbone, Adapter and BitFit update about 6 million and 2 million parameters, respectively, while ATTEMPT updates only 172k parameters per task.ATTEMPT updates only soft prompts and does not modify the LM’s higher layers, resulting in moderate parameter increases compared with other approaches.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate ATTEMPT across diverse target tasks using high-resource source datasets, with comparisons against fine-tuning, prompt-based, adapter, and multi-task baselines. The setup also distinguishes single-task from shared-attention multi-task training and specifies the evaluation metrics.

4.1 Source and Target Tasks

  • The study uses six high-resource sources—MNLI, QNLI, QQP, SST-2, SQuAD, and ReCoRD—and evaluates 21 diverse targets, including scientific entailment, Yelp sentiment, commonsense reasoning, and Wikipedia paraphrase.The target-task set also spans entailment, paraphrase detection, sentiment analysis, and question answering.Source: §4.1 · p. 5

4.2 Baselines and Implementation Details

  • Baselines include fine-tuning, prompt tuning, SPoT, Adapter, AdapterDrop, and BitFit; GLUE comparisons also include multi-task methods such as HyperFormer and AdapterFusion.The GLUE comparisons also include FT-multi-task, Adapter-multi-task, and HyperDecoder.Source: §4.2 · p. 5
  • ATTEMPT uses T5-base; ATTEMPT-m shares attention across target tasks, whereas ATTEMPT trains task-specific attention.Training runs for 20, 10, or 5 epochs according to dataset size; the authors report T5-LM-adapt v1.1 was sensitive and hard to tune, so they use T5.Source: §4.2 · pp. 5–6
  • The GLUE table reports Pearson correlation for STS-B, F1 for MultiRC, accuracy for other tasks, and parameters trained per task; the other-datasets table reports MRQA F1 and accuracy elsewhere.The GLUE comparison excludes SQuAD and ReCoRD from the source-prompt inventory; both tables use T5-base results as specified in the GLUE caption, while the second caption defines its dataset scope.Source: Table 1 · p. 6, Table 2 · p. 6

5 Results

ATTEMPT-m delivers strong task performance with far fewer updated parameters, and few-shot results and analyses examine transfer across source tasks, backbone sizes, and task similarities.

5.1 Main Results

  • ATTEMPT-m beats PT, SPoT, BitFit, and GLUE/SuperGLUE multi-task baselines while matching Adapter or fine-tuning with fewer updated parameters and a frozen LM.It achieves 72.8 average F1 on MRQA, outperforming BitFit using twice as many parameters, and 85.6% average accuracy on WinoGrande, Yelp, SciTail, and PAWS, versus BitFiT (84.7%) and Adapter (86.2%).Source: §5.1 · pp. 6–7

5.2 Few-shot Domain Adaptations

  • ATTEMPT significantly outperforms other methods in most few-shot settings after transfer from GLUE tasks using 4, 16, or 32 examples.The experiments cover BoolQ, CB, and SciTail; the reported outcome is attributed to non-destructive transfer from multiple source tasks.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Analyses

  • ATTEMPT benefits as backbone size increases, matches fully fine-tuned models with T5-base or T5-large, and matches or outperforms Adapter with T5-3B using 37 times fewer parameters.Figure 4 summarizes performance for Adapter, ATTEMPT, prompt tuning, and fine-tuning on three SuperGLUE datasets.Source: §5.3 · p. 7
  • Ablations indicate that target-task prompts, learned attention, and multiple source prompts contribute to performance; constant attention especially hurts BoolQ and NewsQA.The NewsQA ablation uses randomly sampled 10k training examples.Source: §5.3 · p. 8, Table 4 · p. 8
  • Adding source prompts improves BoolQ and RTE except when SQuAD is added to RTE; the authors suggest possible negative transfer from QA prompts to RTE.Adding two QA source prompts helps BoolQ, illustrating the modularity of adding or removing source tasks.Source: §5.3 · p. 8
  • Attention often favors related source tasks, such as Yelp→SST-2 and PAWS-Wiki→QQP, while WinoGrande and SciTail place more attention on target embeddings.The authors note QQP receives high attention from some seemingly different tasks, including MultiRC and WNLI; they hypothesize that MNLI initialization affects attention to RTE.Source: §5.3 · p. 8

6 Related Work

Prior work composes task-specific modules, generates parameters with shared networks, or transfers prompts in other ways. ATTEMPT focuses on transferring reusable soft prompts across tasks without relying on precomputed task-selection priors.

Related parameter-efficient tuning

Multi-task transfer settings

  • Unlike work emphasizing zero- or few-shot transfer without parameter updates, this paper studies transferring source-task knowledge during target-task training.The contrast is with approaches adapting instruction- or demonstration-trained multi-task LMs without parameter updates (Sanh et al., 2022; Min et al., 2021; Wang et al., 2022a; Wei et al., 2022).Source: §6 · p. 9

Modular multi-task training

  • ATTEMPT shares the modular-learning intuition of Ponti et al. (2022), but focuses on efficient knowledge transfer from resource-rich tasks while keeping the LM intact and updating fewer parameters.The cited concurrent work emphasizes few-shot generalization ability.Source: §6 · p. 9

7 Conclusion

ATTEMPT interpolates reusable source-task soft prompts with a new task-specific prompt for each instance while keeping the original LM frozen. Experiments report a trade-off between task performance and efficiency, alongside interpretable and modular transfer.

7 Conclusion

  • ATTEMPT produces instance-wise prompts by interpolating reusable source-task soft prompts with a new task-specific prompt, while keeping the original LM frozen.The conclusion characterizes large-scale experiments as demonstrating a trade-off between task performance and efficiency, with interpretable and modular task transfer.Source: §7 · p. 9

Limitations

The authors identify computational costs, task coverage, source-task coverage, and language coverage as limitations of ATTEMPT’s evaluation and design.

Limitations

  • Adding m prompt tokens increases input length, memory footprint, and computational costs.The authors note that larger backbones may allow shorter prompts.Source: Limitations · p. 9
  • The evaluation covers classification and QA, not tasks requiring long sequence generation such as summarization.The authors suggest exploring more diverse task sets in future work.Source: Limitations · p. 9
  • Experiments use six representative source tasks without exploring large-scale combinations of source tasks.Source: Limitations · p. 9
  • ATTEMPT is not tested on non-English tasks; non-English effectiveness and cross-lingual transfer remain future work.Source: Limitations · p. 10

Ethics Statement

The authors frame ATTEMPT as a way for groups with limited computational resources to benefit from large-scale models, while noting dataset-related risks.

Ethics Statement

  • ATTEMPT aims to help groups with limited computational resources benefit from state-of-the-art large-scale models.Source: Ethics Statement · p. 10
  • The experiments use widely used general-purpose datasets, which the authors consider unlikely to include harmful content.Source: Ethics Statement · p. 10
  • Yelp Review may carry greater privacy or harmful-content risks than datasets sourced from news or encyclopedic websites.Source: Ethics Statement · p. 10

A.1 Improving Multi-task Training

The appendix describes training strategies for ATTEMPT, its parameter scaling, and an alternative token-level attention design. It reports that ATTEMPT’s trainable parameters increase less rapidly with backbone size than other parameter-efficient approaches.

A.1 Improving Multi-task Training

  • Learning prompt interpolations is difficult without meaningful embedding-token correspondences or ground-truth task mappings, so the authors explore inductive-bias strategies.They pretrain an attention prior on source tasks for target-task initialization or tasks without training data, and also introduce different learning rates for the composition module and task-specific parameters.Source: §A.1 · p. 14

A.2 Overview of Training

  • Table 5 presents an overview of ATTEMPT’s training process.Source: §A.2 · p. 14

A.3 Parameter Efficiency of ATTEMPT

  • ATTEMPT’s trainable-parameter count increases only slightly as backbone size grows, unlike other parameter-efficient approaches.It keeps the original LM frozen and does not modify LM structures.Source: Figure 7 · p. 14, §A.3 · p. 14

A.4 Alternative Attention Design

  • Token-level attention scores each prompt token separately and forms each summary from a weighted combination of corresponding prompt tokens.It performs similarly to the default attention and improves some tasks, but adds computational overhead; the authors therefore retain max-pooling-based unified attention as default.Source: §A.4 · p. 14

B Task and Dataset Details

The appendix documents dataset scope, implementation and prompt-training choices, hyperparameters, multi-task grouping, and few-shot adaptation procedures.

B Task and Dataset Details

  • The appendix lists six source tasks and 17 target tasks, with task, domain, and output-format diversity including span extraction, multiple choice, and classification.Tables 6 and 7 provide source- and target-task details; four GLUE datasets also serve as evaluation targets.Source: B Task and Dataset Details · p. 14, Table 6 · p. 16, Table 7 · p. 16

C.1 Implementation Details

  • The models use PyTorch and Hugging Face Transformers, with baseline implementations and datasets drawn from cited implementations and repositories.MRQA 2019 training and development data are downloaded from its official repository.Source: §C.1 · p. 15

C.2 Source Prompt Training Details

  • Each source prompt is trained for five epochs from randomly sampled tokens, and the checkpoint with the best development score is retained.The authors report reasonable performance from this initialization on the six source tasks, including with T5-small.Source: §C.2 · p. 15

C.3 Attention Module Pretraining Details

  • Attention-module pretraining splits source tasks by input-context length, using the SQuAD/ReCoRD group’s weights for MRQA and the other group’s weights elsewhere.The shorter-context group comprises MNLI, SST-2, QQP, and QNLI.Source: §C.3 · p. 15

C.4 General hyperparameters

  • Experiments use dataset-specific maximum input lengths, one 24-GB GPU, and complete training within 24 hours.Maximum lengths are 512 tokens for MRQA, 348 for MultiRC, and 256 for other datasets; MRQA uses batch size 16 with gradient accumulation of 2 due to out-of-memory errors.Source: §C.4 · p. 15

C.6 Hyperparameters for Baselines

  • Baseline settings include 500 warmup steps and Adam; SPoT-targeted selects the top similar source prompt, while Adapter uses reduction factor 32 and learning rate 3 × 10^-4.SPoT-g was not strong on MRQA or several other task sets, so the main experiments use SPoT-t.Source: §C.6 · p. 16

C.7 Multi-task Training Details

  • Multi-task training groups four dataset sets, trains ATTEMPT-m on eight GLUE tasks, and keeps MultiRC separate because of its longer context.Source: §C.7 · p. 17

C.8 Few-shot Adaptation Experiments Details

  • Few-shot adaptation is run three times and averaged; ATTEMPT loads attention weights trained on eight GLUE tasks.Fine-tuning, Adapter, and HyperFormer results are taken from Mahabadi et al. (2021b).Source: §C.8 · p. 17

C.9 Scaling Experiments Details

The scaling experiments show that, under a 24-GB GPU constraint, ATTEMPT can train with T5-3B while full fine-tuning cannot. The memory-footprint analysis finds a moderate increase for ATTEMPT as backbone size grows, alongside substantially higher footprints for Adapter and BitFit relative to T5-base.

C.9 Scaling Experiments Details

  • Under a 24-GB GPU constraint, ATTEMPT, Adapter, and prompt tuning can train with T5-3B, but full fine-tuning cannot, even at batch size 1.For Adapter and fine-tuning, GPU memory usage was restricted to 24 GB; scalability experiments used a maximum token length of 216.Source: §C.9 · p. 17
  • For ATTEMPT and prompt tuning, per-GPU batch sizes were 100 and 32 for T5-small and base, and 16 and 2 for T5-large and T5-3B, respectively.Adapter and fine-tuning batch sizes were adjusted by dataset to avoid out-of-memory errors where possible.Source: §C.9 · p. 17
  • T5-large fine-tuning is occasionally unstable, fails to learn a target task in some runs, and is sensitive to batch size and learning rate.Different batch sizes yielded 65% BoolQ accuracy; the reported results use the best performance found after exploring learning rates and batch sizes.Source: §C.9 · p. 17

C.10 Memory Footprints

  • ATTEMPT’s memory footprint rises from 13.7 GB to 16.1 GB as the backbone grows, while Adapter and BitFit use about three times more memory than T5-base.The paper attributes ATTEMPT’s increased footprint to longer inputs, multiple pre-loaded source prompts, and attention calculations; prompt length may be reduced as backbones grow.Source: §C.10 · pp. 17–18, Table 9 · p. 18

Related papers

Loading 2205.11961v2…