PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson

arXiv:2106.04489v1cs.CL

TL;DR

The paper addresses the limited cross-task sharing of separately trained adapters in parameter-efficient fine-tuning. It uses shared hypernetworks conditioned on task, layer, and adapter position to generate task-specific adapters, achieving strong GLUE and domain-generalization results with efficient adaptation.

  • Problem

    Separately trained adapter modules do not share information across tasks, while multi-task fine-tuning can suffer from constrained capacity and negative transfer.

  • Method

    HYPERFORMER++ uses hypernetworks shared across tasks and layers to generate task-specific adapter and layer-normalization parameters from learned task-conditioned representations.

  • Results

    The method achieves strong gains over adapters on GLUE, including 82.51 versus 79.53 for T5SMALL and 86.48 versus 84.88 for T5BASE.

  • Takeaways & Limitations

    Shared hypernetworks capture knowledge across tasks while task-specific layers support adaptation, improving multi-task learning and in-domain task generalization.

  • Takeaways & Limitations

    The experiments did not test more complex sampling strategies or tuning of T.

Abstract

from arXiv · show

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.

1 Introduction

HYPERFORMER++ addresses the tension between multi-task knowledge sharing and task-specific adaptation by generating adapters with shared hypernetworks. It reports strong GLUE gains while reducing per-task parameter costs.

  • Multi-task fine-tuning can share information and improve low-resource tasks, but constrained capacity and task interference can hurt performance.
  • Adapters preserve the pretrained model while isolating task-specific information, but prior adapters lack an effective parameter-efficient mechanism for cross-task sharing.
  • HYPERFORMER++ shares a compact hypernetwork across tasks and layers to generate task- and layer-specific adapter parameters from task and layer embeddings.
  • The method aims to share information across tasks while minimizing negative interference through separate task-specific adapter layers.
  • Strong gains are reported over T5BASE and Houlsby adapters on GLUE, demonstrating effective information sharing while minimizing negative transfer.
  • The paper reports parameter-efficient multi-task fine-tuning and extensive few-shot domain-transfer experiments showing positive transfer to unseen in-domain tasks.

2 HYPERFORMER

HYPERFORMER integrates task-conditioned adapter layers and layer normalization into a multi-task Transformer using hypernetworks. Its shared hypernetworks capture cross-task information while task-specific generated parameters preserve individual task adaptation and reduce interference.

  • Task-conditioned adapters: HYPERFORMER generates task-specific adapter parameters from task embeddings using hypernetworks, while keeping the pretrained model parameters fixed except selected components.Training updates hypernetworks, task embeddings, and layer normalizations; the remaining pretrained parameters stay fixed.
  • Task-conditioned adapters: Hypernetworks capture shared information across tasks, while generated task-conditioned adapters and layer normalization support task adaptation and reduce negative interference.This combines cross-task sharing with task-specific parameterization.
  • Task conditional adapter layers: Conditional adapters are inserted after each attention and feed-forward block before the Transformer skip connection.The adapter uses a bottleneck dimension d, with down-projection, GeLU, and up-projection components.
  • Task representations: Task embeddings are produced by a projector from learnable task parameters or pretrained task features, and T5 task prefixes are replaced with embedding-based conditioning.In this work, learnable task parameters enable end-to-end training.
  • Task representations: Pilot experiments found no extra benefit from pretrained task embeddings, although the framework permits either learnable parameters or pretrained task features.The reported pilot result concerns pretrained task embeddings specifically.
  • HYPERFORMER++: HYPERFORMER++ shares hypernetworks across Transformer layers, using task, layer-id, and adapter-position embeddings to produce distinct parameters for each location.Sharing reduces parameters, while learned layer and position embeddings preserve flexible parameterization.

3 Experiments

Experiments evaluate HYPERFORMER++ on GLUE, low-resource training, and few-shot domain transfer. The method improves performance over adapter and fine-tuning baselines while using fewer trainable parameters.

  • 3.1 Results on the GLUE Benchmark: GLUE experiments compare HYPERFORMER++ with adapter baselines and T5 models across paraphrase, sentiment, inference, and acceptability tasks.The evaluation uses standard GLUE tasks and reports task-appropriate metrics.
  • 3.1 Results on the GLUE Benchmark: 82.51 versus 79.53 for T5SMALL and 86.48 versus 84.88 for T5BASE shows HYPERFORMER++ gains over Adapters while being more parameter-efficient.
  • 3.1 Results on the GLUE Benchmark: 0.45 for T5SMALL and 1.81 for T5BASE are the average improvements over single-task fine-tuning, with larger gains on low-resource CoLA and RTE.CoLA improves from 54.85 to 63.73, while RTE improves from 67.39 to 75.36.
  • 3.1 Results on the GLUE Benchmark: 0.29% trainable parameters per task lets HYPERFORMER++BASE use 3× fewer trainable parameters than AdaptersBASE.Its total parameter count is 1.02× that of T5BASE.
  • 3.4 Few-shot Domain Transfer: Few-shot domain transfer evaluates five tasks across seven datasets, and HYPERFORMER++ significantly surpasses baselines on the majority of settings.The evaluation uses models trained on GLUE and reports results after few-shot fine-tuning on target data.
  • 3.3 Low-resource Fine-tuning: HYPERFORMER++ substantially improves GLUE results with limited training data, indicating more effective fine-tuning in the low-resource regime.Figure 2 evaluates 100, 500, 1000, 2000, and 4000 samples per task across five seeds.

4 Analysis

HYPERFORMER++ is more parameter-efficient than adapters because its hypernetwork parameters remain constant while task features scale more slowly. Ablations and task-embedding analyses further examine whether its gains arise from shared information and whether learned embeddings reflect task similarity.

  • Parameter Efficiency: HYPERFORMER++ hypernetwork parameters remain constant, while task-feature parameters scale with the number of tasks or layers.The task-feature embedding size is t=64 in the experiments.
  • Parameter Efficiency: HYPERFORMER++ is much more parameter-efficient than adapters when the numbers of layers and tasks are large.This follows because t≪2hd+2h and T+L≪TL.
  • Extra Parameters: Even with more parameters, Adapters† cannot reach HYPERFORMER's performance on GLUE.The comparison tests whether HYPERFORMER's improvements are attributable to information sharing rather than parameter count.
  • Ablations: Each evaluated component contributes positively to the framework's average GLUE performance.The ablation removes task-conditional adapters, layer normalization, projection networks, and related fine-tuning components.
  • Task Embeddings: PCA projections show that learned task embeddings group similar tasks, including CB near RTE after few-shot training.The CB–RTE grouping is attributed to greater premise and hypothesis complexity than in MNLI.

5 Related Work

Prior work addresses multi-task learning through unified models, task-specific adapters, hypernetworks, distillation, and contextual parameter generation. HYPERFORMER++ differs by generating small task-specific adapter parameters across multiple tasks and layers with shared hypernetworks.

  • Multi-task learning: Multi-task learning seeks one model that performs well across different NLP tasks but faces catastrophic forgetting and disproportionate task sizes.These imbalances can produce overfitting on low-resource tasks and underfitting on high-resource tasks.
  • Multi-task learning: MTDNN learns from multiple NLU tasks but uses multi-task learning as pretraining followed by task-specific fine-tuning.The passage contrasts this setup with unified multi-task fine-tuning.
  • Related approaches: Tay et al.'s task-conditioned hypernetworks are reported as 43x less parameter-efficient than HYPERFORMER++.Other prior approaches include knowledge distillation and separately trained or task-specific adapters.
  • Hypernetworks: Unlike continual-learning hypernetworks that generate all target-model weights, HYPERFORMER++ generates only a small number of adapter parameters.This allows adaptation while avoiding generation of the full target model.
  • Contextual parameter generation: Meta-learning and Bayesian approaches that generate softmax parameters are described as slower to train and restricted to classification settings.The passage also states that they cannot adapt lower model layers through generated softmax parameters.

6 Conclusion

HYPERFORMER++ uses shared hypernetworks to generate task-specific adapters conditioned on task, layer, and adapter position. The authors report strong GLUE improvements and substantially better in-domain task generalization while combining shared knowledge with task adaptation.

  • Method: HYPERFORMER++ trains shared hypernetworks to generate task-specific adapters conditioned on task, layer id, and adapter position embeddings.The shared hypernetworks capture knowledge across tasks while task-specific layers preserve individual-task adaptation.
  • Results: The method obtains strong improvement over multi-task learning on GLUE and substantially improves in-domain task generalization.The conclusion presents these as outcomes of extensive experiments.

A Experimental Details

The experiments evaluate efficiency and adapter bottleneck choices on GLUE under specified hardware, preprocessing, and training settings. HYPERFORMER++ uses less memory overhead and incurs a smaller training-time increase than the compared adapter baseline, while reduction factors should be tuned per dataset.

  • Infrastructure: Experiments use 4 GPUs for Table 1 and 1 GPU for the remaining experiments on a heterogeneous GPU cluster.The cluster includes Tesla V100, Tesla A100, Tesla P4, and GTX1080ti GPUs.
  • Hyperparameters: The experiments use task-feature dimension t′=512, task-embedding dimension t=64, and reduction factors {16,32,64} for low-resource fine-tuning.Batch sizes are 64 for T5SMALL and 32 for T5BASE.
  • Preprocessing: Datasets are cast into sequence-to-sequence format, with STS-B recast as a 21-class classification task.STS-B target scores are rounded to the nearest increment of 0.2.
  • Evaluation: Efficiency evaluation reports memory and the time required for 1000 training steps on GLUE.Tables 5 and 6 compare these quantities across methods.
  • Efficiency: 30.49%: HYPERFORMER++BASE's training-time increase relative to T5BASE, versus 84.93% for Adapters†BASE.The passage also reports much better memory usage with low overhead for HYPERFORMER++BASE.
  • Efficiency Tables: Table 5 reports required memory for all methods, with percentage differences measured relative to T5BASE.The table is part of the GLUE efficiency evaluation.
  • Bottleneck Size: Adapter reduction factors need to be set per dataset, with smaller datasets generally benefiting from smaller bottleneck sizes.Larger datasets generally require more modeling capacity and larger bottleneck sizes.
  • Bottleneck Size: Table 7 reports GLUE validation performance for reduction factors r={8,16,32} using task-specific metrics.Reported metrics include accuracy, F1, Pearson and Spearman correlations, and Matthews correlation.
Loading 2106.04489v1…