Source-linked AI summary
AdapterHub: A Framework for Adapting Transformers
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
TL;DR
Large pretrained models are costly to store and share, while integrating independently trained adapters is not straightforward. AdapterHub provides tooling and infrastructure for training, sharing, downloading, and stitching adapters into pretrained transformers, with reported efficiency and reproducibility benefits. The framework’s stated scope includes task and language adaptation, although mapping adapters between different models remains future work.
Problem
Storing and sharing separately fine-tuned large pretrained models is expensive, while reusing adapters is complicated by differing architectures and model, task, and language dependencies.
Method
AdapterHub extends HuggingFace Transformers with trainable adapter modules, automatic extraction and separate storage, online sharing, downloading, and adapter composition.
Results
AdapterHub enables lightweight adaptation and sharing across tasks and languages, requiring as little as two added code lines and offering adapters that can require as little as 1Mb of storage.
Takeaways & Limitations
AdapterHub makes task-, domain-, and language-specific models easier to access and supports more efficient, reproducible transfer learning and adapter composition.
Takeaways & Limitations
Mapping adapters between different models is identified as future work.
Abstract
from arXiv · showhide
The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters -- small learnt bottleneck layers inserted within each layer of a pre-trained model -- ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages. The framework, built on top of the popular HuggingFace Transformers library, enables extremely easy and quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Downloading, sharing, and training adapters is as seamless as possible using minimal changes to the training scripts and a specialized infrastructure. Our framework enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. AdapterHub includes all recent adapter architectures and can be found at https://AdapterHub.ml.
1 Introduction
Large pretrained transformers achieve strong NLP performance, but storing and sharing separately fine-tuned models is often prohibitive. AdapterHub addresses the difficulty of reusing adapters by providing a framework for training, sharing, and combining them.
- Transformer language models are pretrained on large text corpora, fine-tuned for target tasks, and achieve state-of-the-art performance across many language-understanding tasks.Recent models have reached billions of parameters.
- Training and sharing separately fine-tuned large models is often prohibitive, limiting modular architectures, task composition, and injection of external information.
- Adapters train small, newly initialized layers while keeping the pretrained model’s parameters fixed, enabling task-specific and language-specific modules for one shared model.These modules can be exchanged and combined after training.
- AdapterHub makes adapter reuse easier by addressing their fragmented release, architectural differences, and dependence on model, task, and language.The framework is designed to facilitate transfer learning across settings.
- Built on HuggingFace Transformers, AdapterHub supports adapter integration with minimal code edits and provides a website for uploading, downloading, and sharing pretrained adapters.The framework is available online at AdapterHub.ml.
- AdapterHub supports transformer models including BERT, RoBERTa, and XLM(-R), stores adapter weights separately, and provides adapter composition and stacking.Training adapters may require as little as two additional lines of code, while adapter storage can require as little as 1Mb.
2 Adapters
Adapters add small trainable modules to frozen pretrained transformers, with architecture and placement choices affecting their implementation. They provide lightweight task adaptation, substantially smaller task-specific storage, and modular composition.
- Adapter Architecture: Adapters introduce a small parameter set Φ into each transformer layer while keeping the pretrained parameters Θ fixed during target-task training.The trainable parameters encode task-specific representations in intermediate layers.
- Adapter Architecture: Adapter placement and architecture vary across implementations, although two-layer feed-forward bottleneck networks are empirically supported.LayerNorm additions and placement within transformer blocks are among the differing design choices.
- Task-specific Layer-wise Representation Learning: Adapters have no large performance difference from full fine-tuning across the reported GLUE settings, supporting them as a lightweight alternative for state-of-the-art downstream performance.Table 1 compares full fine-tuning with Pfeiffer and Houlsby adapter architectures using BERT-Base weights and bottleneck size 48.
- Small, Scalable, Shareable: 2.2Gb of compressed storage is needed for XLM-R Large, while a single adapter can require as little as 0.9Mb for one task.Full fine-tuning otherwise requires storing a separate copy of the model for each task.
- Small, Scalable, Shareable: For BERT-Base, storing 2 fully fine-tuned models uses the same space as 125 adapter models with bottleneck size 48 and Pfeiffer adapters.More than 99% of the parameters required for each target task remain fixed and can be shared for inference.
- Modularity of Representations: Adapters can be stacked or dynamically replaced because their encapsulated modules learn output representations compatible with subsequent transformer layers.Adapters independently trained for specific tasks and languages have been successfully combined.
3 AdapterHub
AdapterHub supports the full adapter lifecycle, from adding and training frozen-base-model adapters to extracting, sharing, finding, and stitching them into transformers for inference.
- AdapterHub combines a HuggingFace-based library with a website for managing adapters across their full lifecycle.The lifecycle includes introducing and training adapters, extracting and open-sourcing them, visualizing configurations, downloading them, and using them for inference.
- Two additional lines of code add adapter weights and freeze the pre-trained transformer while training proceeds on the adapter and prediction head.The framework supports task and language adapters and allows other adapter architectures to be selected.
- Only adapter weights and the prediction head need to be stored, while the unchanged base model remains shared.Automatic extraction reduces storage requirements and permits many checkpoints to be stored simultaneously.
- AdapterHub dynamically configures adapter placements, residual connections, LayerNorms, bottleneck sizes, and activation functions, while supplying literature-based presets.Preset configurations include architectures proposed by Pfeiffer et al. and Houlsby et al., with an additional configuration for Bapna and Firat.
- The website organizes adapters by tasks, languages, models, and datasets, then provides compatible choices and metadata for selection.Adapters depend on the transformer model on which they were trained and are otherwise incompatible.
- A selected pre-trained adapter can be downloaded and stitched into the transformer with a single line of code for inference.The website supplies sample code corresponding to the configuration required for the selected adapter.
- Adapters currently depend on the transformer model used during training, while mapping adapters between different models remains future work.
- Adapters are usable for their designated tasks and can also support transfer, stacking, fusion, and extensions to other modalities.The paper presents these uses as further research directions and possible modes of usage.
4 Conclusion and Future Work
AdapterHub provides a simple framework for training and community sharing of adapters in pre-trained transformers. It emphasizes efficient, reproducible, and extensible transfer learning, with continued plans for composition and broader model support.
- AdapterHub enables simple and effective transfer learning by training and community sharing of adapters.
- Adapters can be stitched into large pre-trained transformers across languages and tasks, requiring as little as two lines of code in existing scripts.
- The framework offers improved reproducibility, greater efficiency than full fine-tuning, extensibility to new models and tasks, and access to trained models.
- Future work includes improving composition and modularity, extending the framework, and supporting additional transformer models.