PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Universal Intrinsic Task Subspace via Prompt Tuning

Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, Jie Zhou

arXiv:2110.07867v3cs.CLcs.AI

TL;DR

The paper asks why PLMs adapt to diverse NLP tasks with little data and proposes IPT to find a shared low-dimensional subspace of task prompts. In a 250-dimensional subspace found using 100 tasks, tuning 250 parameters recovers 97% of full prompt-tuning performance on seen tasks and 83% on unseen tasks.

  • Problem

    The mechanisms behind PLMs’ adaptation to diverse downstream tasks remain unclear, including how they adapt across tasks with small-scale data.

    Source: §1 · p. 1

  • Method

    IPT learns a shared nonlinear subspace from multiple tasks’ soft prompts, then adapts to unseen data or tasks by tuning parameters within that subspace.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    A 250-dimensional subspace found with 100 tasks recovers 97% of full prompt-tuning performance for 100 seen tasks and 83% for 20 unseen tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results provide evidence that PLM adaptations across tasks can be reparameterized within a shared low-dimensional subspace.

    Source: §5 · p. 8

    IPT’s effectiveness could still be improved, especially at low dimensions.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Why can pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot superficially? In this work, we empirically find evidence indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a unified low-dimensional intrinsic task subspace, which may help us understand why PLMs could easily adapt to various NLP tasks with small-scale data. To find such a subspace and examine its universality, we propose an analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort to the recent success of prompt tuning and decompose the soft prompts of multiple NLP tasks into the same low-dimensional nonlinear subspace, then we learn to adapt the PLM to unseen data or tasks by only tuning parameters in this subspace. In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace. Besides being an analysis tool, IPT could further help us improve the prompt tuning stability.

1 Introduction

The paper asks why PLMs adapt to superficially diverse tasks and hypothesizes that their adaptations share a low-dimensional intrinsic task subspace. It introduces IPT to find and test that subspace, reporting recovery of much of full prompt-tuning performance on seen and unseen tasks.

1 Introduction

  • The paper hypothesizes that diverse PLM adaptations can be reparameterized by optimizing a few free parameters in one low-dimensional intrinsic task subspace.The hypothesis is motivated by the unresolved mechanisms behind adaptation across downstream tasks.Source: §1 · p. 1
  • A shared subspace, if it exists, could explain PLM universality, while its low dimensionality could explain adaptation with small-scale data.The paper frames PLMs as compressing task-learning complexity from high to low dimensionality.Source: §1 · p. 2
  • Prompt tuning makes subspace analysis computationally tractable because its adaptive soft prompts number only tens of thousands, unlike all PLM parameters.The passage also notes prompt tuning can approach fine-tuning performance on understanding and generation tasks.Source: §1 · p. 2
  • IPT uses multi-task subspace finding to decompose task soft prompts, then intrinsic subspace tuning to adapt on unseen data or tasks through a fixed back-projection.The two phases are MSF and IST.Source: §1 · p. 2
  • A 250-dimensional subspace found from 100 tasks recovers 97% of full prompt-tuning performance on 100 seen tasks and 83% on 20 unseen tasks.The results use only 250 free parameters for adaptation.Source: §1 · p. 2
  • The paper also reports that IPT and the subspace could help analyze task differences and improve prompt-tuning stability.It encourages future work on better subspace finding and techniques inspired by universal reparameterizations.Source: §1 · p. 2

2 Related Work

Prior work shows that PLMs can be adapted through fine-tuning or prompt-based methods, while studies of intrinsic dimensionality motivate examining whether task adaptations share a subspace. This paper uses a nonlinear, multi-task approach to investigate that possibility.

2 Related Work

  • Prompt tuning freezes the PLM and tunes task-specific soft prompts, building on prompt methods that cast tasks into pre-training-like forms.Earlier work found soft prompts can perform well across tasks, especially with large PLMs.Source: §2 · p. 2
  • Intrinsic dimension is the minimum number of variables needed to represent data or approximate a function; prior work found low-dimensional PLM adaptation for individual tasks.Aghajanyan et al. (2021) report adaptation dimensions below thousands and that pre-training implicitly lowers them.Source: §2 · p. 2
  • Unlike random linear projections, this work uses supervision from diverse tasks to train a nonlinear low-dimensional decomposition intended to reduce task-irrelevant redundancy.The target question is whether individual task subspaces can be universal.Source: §2 · p. 3
  • The paper situates its analysis among efforts to unify superficially different NLP tasks, including approaches that cast them into PLM pre-training formats.Its analysis concerns task unification from the perspective of intrinsic task subspaces.Source: §2 · p. 3

3 Methodology

The methodology frames NLP tasks as conditional generation and contrasts full-parameter fine-tuning with prompt tuning. IPT then finds a shared nonlinear prompt subspace across tasks and evaluates whether tuning within it transfers to unseen data and tasks.

3 Methodology

  • The methodology introduces fine-tuning, prompt tuning, and IPT as the adaptation paradigms, with IPT divided into MSF and IST.Figure 2 contrasts the paradigms and their parameter roles.Source: §3 · p. 3

3.1 Preliminaries

  • Tasks are cast as conditional generation, where a PLM maps input tokens to target-token sequences.The model embeds and encodes the input, then decodes the target conditioned on hidden representations.Source: §3.1 · p. 3
  • Prompt tuning prepends task-specific soft-prompt embeddings to the input, freezes PLM parameters, and tunes only the prompts.Traditional fine-tuning instead tunes all PLM parameters.Source: §3.1 · p. 3

3.2 Intrinsic Prompt Tuning

  • MSF learns a projection and back-projection from multiple tasks’ trained prompts, optimizing reconstruction and task-oriented losses to find a shared subspace.The projection is a one-layer feed-forward network and the back-projection a two-layer nonlinear perceptron.Source: §3.2 · p. 4
  • MSF trains only the projection and back-projection parameters, using task supervision and nonlinearity to seek more effective, less redundant subspaces than random linear subspaces.The objective balances reconstruction and task-oriented losses with a hyperparameter.Source: §3.2 · p. 4
  • IST tests transfer to unseen training data and tasks by freezing the PLM and learned back-projection, then tuning only the intrinsic vector’s bottleneck-dimension parameters.The vector is projected into soft prompts through the fixed back-projection.Source: §3.2 · pp. 3–4

4 Experiment and Analysis

Across 120 few-shot tasks, the experiments test IPT’s subspace across seen and unseen data and tasks, then analyze task-type effects and other properties. At 250 dimensions, IPT recovers over 90% of full prompt-tuning performance on unseen data and 83% on unseen tasks.

4.1 Experimental Settings

  • The evaluation uses 120 few-shot NLP tasks across three splits, with training and development sizes balanced by task category to limit bias toward data-rich tasks.Classification and regression use 16 examples, while other categories use 32.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Main Results

  • On unseen i.i.d. data, IST recovered more than 90% of full prompt-tuning performance of the 100 training tasks; on unseen tasks, tuning 250 parameters achieved 83% performance, supporting generalization.Source: §4.2 · p. 5
  • MSF reconstruction performs poorly on unseen prompts, while IST finds better unseen-task solutions; optimization in IST also fails to reach good same-data solutions available in the subspace.The authors identify auto-encoder reconstruction capacity and the adopted optimization algorithm as possible constraints.Source: §4.2 · p. 6
  • Subspaces trained on one task type work well on that type but generalize poorly to the other, revealing a substantial classification versus non-classification gap.The authors attribute this pattern to bias in the found subspace toward training task types.Source: §4.2 · p. 6

4.3 Analyses and Properties

  • Random subspaces can support effective reparameterization at large dimensions, but IPT performs better with fewer dimensions; adapter-based IPT performs consistently better than prompt-based IPT.Adapter-based IPT can outperform fine-tuning at 2000 dimensions.Source: §4.3 · p. 7, Figure 4 · p. 7
  • As the data scale grows, IPT performance generally improves, especially at low dimensions, consistent with the authors’ observation that more data yields a more universal subspace.The analysis doubles or quadruples the few-shot examples for seen and unseen tasks.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Increasing the number of training tasks generally improves generalization to unseen tasks, which the authors link to broader coverage and diversity of seen tasks.The analysis varies the training-task sample from 20% to 80%.Source: §4.3 · p. 7, Figure 6 · p. 8
  • PCA separates classification from non-classification task vectors while mixing seen and unseen task vectors; IPT is also reported as the most stable method, and its initialization lowers prompt-tuning variability.With IPT initialization, standard deviation is 1.65 versus 4.19 for vanilla prompt tuning, while relative performance reaches 103.4%.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

5 Conclusion and Discussion

The paper concludes that PLM adaptations can be optimized within a unified low-dimensional intrinsic task subspace, and presents IPT as an analysis tool for testing this hypothesis. It also connects subspace characteristics to questions about scaling, task relationships, and practical tuning methods.

5 Conclusion and Discussion

  • IPT jointly decomposes multiple tasks’ adaptive parameters and tunes within the resulting subspace, providing evidence for the hypothesis that PLM adaptations can be reparameterized there.The authors conclude that the found subspaces contain good solutions for PLM adaptations.Source: §5 · p. 8
  • The authors suggest that larger PLMs may be more easily reparameterized into unified subspaces, potentially helping explain their cross-task generalization and lower data and parameter needs.They note that larger PLMs have lower reparameterization dimensions (Aghajanyan et al., 2021).Source: §5 · p. 8
  • The intrinsic vectors obtained during IST may help identify task relationships and support investigation of cross-task generalization.The authors describe these as future research directions.Source: §5 · p. 8
  • The authors encourage further work on low-dimensional subspace tuning, which may avoid overparameterization and use fewer tunable parameters.They also describe this as potentially more environmentally friendly.Source: §5 · p. 9

A Additional Experiments

The appendix adds absolute-performance reporting and further checks relative performance against fine-tuning. BARTLARGE results broadly align with BARTBASE, while performance is worse at extremely low dimensions, especially for classification tasks.

A.1 Absolute Performance

  • The appendix additionally reports average absolute performance, with tables covering prompt tuning and fine-tuning and IPT across the adopted task splits.The IPT tables cover the random, non-cls, and cls splits.Source: §A.1 · p. 17, Table 3 · p. 17, Table 4 · p. 17

A.2 Relative Performance to Fine-tuning

  • Figure 8 reports additional fine-tuning-relative performance for non-cls and cls splits, with conclusions consistent with the analyses in § 4.2.Relative performance to prompt tuning remains the main evaluation metric, except for Figure 3(b).Source: §A.2 · p. 17

A.3 BARTLARGE Performance

  • BARTLARGE results suggest that non-trivial performance can be recovered in the found subspaces, but performance is worse at dimensions 5–10, especially on the cls split.The authors report that their experiments focus on BARTBASE and do not test other PLMs or extremely large models.Source: §A.3 · p. 17

A.4 Comparison of Subspace-Finding Methods on Ttrain

On Ttrain, IPT is compared with alternative subspace-finding methods on the random task split. Random-subspace methods perform worse, especially at low dimensions, while IPT variants remain consistently strong.

A.4 Comparison of Subspace-Finding Methods on Ttrain

  • The comparison evaluates subspace-finding methods on Ttrain, the unseen-data challenge, using the random task split.Figure 10 is identified as the comparison visual.Source: §A.4 · p. 17
  • Randomprompt and Randomall param. perform worse than IPT, especially at low dimensions, while IPT(Prompt) and IPT(Adapter) remain consistently strong across dimensions.The passage describes the random-subspace methods as not involving training and says IPT's training-task performance suggests it fits the reparameterization of those tasks.Source: §A.4 · pp. 17–18
  • IPT(Prompt) and IPT(Adapter) maintain consistently high performance across the tested dimensions.The passage interprets this pattern as evidence that the pipeline can fit the reparameterization of training tasks.Source: §A.4 · p. 18

B Additional Visualization

The additional visualization finds category-level clustering among intrinsic vectors for QA and text-classification tasks. The notations section explains terms used in the paper to help readers navigate it.

B Additional Visualization

  • PCA plots show intrinsic vectors from the same fine-grained QA or text-classification category forming compact clusters.The visualization uses the random task split and intrinsic dimension dI = 100.Source: B Additional Visualization · p. 18
  • The authors propose that intrinsic vectors can serve as task representations for analyzing similarities and differences among NLP tasks.Source: B Additional Visualization · p. 18

C Descriptions of Notations

D Implementation Details

The implementation uses a unified text-to-text setup and specifies the prompt and adapter implementations, training procedures, and auxiliary visualizations.

D Implementation Details

  • All tasks are converted to a unified sequence-to-sequence format for use with text-to-text PLMs.Figure 9 compares IPT with BARTLARGE at different intrinsic dimensions against prompt tuning; Figure 10 compares IPT with other subspace-finding methods on Ttrain for the random task split.Source: D Implementation Details · p. 18, Figure 9 · p. 18, Figure 10 · p. 18

D.1 IPT Details

  • The IPT experiments use AdamW, grid-search learning rates and batch sizes, select checkpoints on Ddev, and evaluate on Dtest.The setup uses an NVIDIA 32GB V100 GPU, sets α to 200, and uses IST hyperparameters matching prompt tuning; the reported MSF and IST runtimes are within 24 hours on 8 GPUs and 6 hours per experiment on 1 GPU, respectively.Source: §D.1 · pp. 18–19
  • The prompt back-projection Proj_b is a two-layer perceptron with trainable weights and biases, and the intrinsic dimension dI is varied.Its inner hidden size is 768 for both BARTBASE and BARTLARGE; Figure 11 visualizes intrinsic vectors by task category.Source: §D.1 · p. 19, Figure 11 · p. 19

D.2 Subspace-Finding with Adapter

  • Adapters add residual feed-forward modules between Transformer layers, using down-projection, activation, and up-projection operations.In MSF, both adapter matrices are reparameterized using a low-dimensional intrinsic vector and projection matrices; during IST, only a randomly initialized dI-dimensional intrinsic vector is optimized.Source: §D.2 · pp. 19–20
  • Figure 12 groups IPT relative performance by fine-grained task type and distinguishes settings where MSF includes or excludes the investigated type.In both settings, IST is tested on tasks of the investigated type.Source: Figure 12 · p. 20, §D.2 · p. 20

E Fine-grained Analyses on Task Types

The fine-grained analysis compares IPT with prompt tuning across task types and examines how training-task composition relates to performance. It reports differences in task-type separation and improvements on some types.

E Fine-grained Analyses on Task Types

  • The analysis covers six classification and six non-classification fine-grained task types, using included and excluded MSF training-task settings.The excluded setting removes the investigated type from MSF training, while IST is run on that type in both settings.Source: E Fine-grained Analyses on Task Types · p. 20
  • Excluding cls/topic, cls/fact checking, or cls/NLI tasks from MSF is associated with a large performance gap versus including them.The authors suggest these types may require skills distinct from other types.Source: E Fine-grained Analyses on Task Types · p. 20
  • Other/linguistic phenomenon, cg/dialogue, and qa/MRC show close performance between included and excluded settings.The authors interpret this as suggesting these types require language skills similar to those of other task types.Source: E Fine-grained Analyses on Task Types · p. 20
  • IPT improves over vanilla prompt tuning on some task types, including cls/topic and qa/MRC.The authors state that these gains suggest promise for tuning PLMs within the intrinsic task subspace.Source: E Fine-grained Analyses on Task Types · p. 20
  • The authors propose that intrinsic dimensions may be viewed as latent skills and raise questions about composing tasks from smaller “atom tasks.”They present these as questions for future research, not established findings.Source: E Fine-grained Analyses on Task Types · p. 21

F Task Details

Table 8 catalogs the evaluated tasks by ontology, task name, and reference, using the task ontology of Ye et al. (2021).

F Task Details

  • Table 8 organizes the evaluated task inventory under ontology, task name, and reference headings.The table refers readers to Ye et al. (2021) for its task ontology.Source: F Task Details · pp. 21, 23, Table 8 · p. 22
  • Other entries include ai2_arc, commonsense_qa, dream, race-high, adversarialqa, quoref, and superglue-record.The table also lists eli5-askh, multi_news, empathetic_dialogues, spider, and blimp-anaphor_gender_agreement.Source: F Task Details · p. 23
  • The inventory includes blimp-anaphor_gender_agreement as a linguistic-phenomenon task.Source: F Task Details · p. 23
  • Listed tasks include blimp-ellipsis_n_bar_1, cos_e, ade_corpus_v2-dosage, kilt_ay2, acronym_identification, and break-QDMR.The inventory also names art and common_gen.Source: F Task Details · p. 23

Related papers

Loading 2110.07867v3…