Source-linked AI summary
AdapterDrop: On the Efficiency of Adapters in Transformers
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych
TL;DR
Large transformer models are expensive to fine-tune, slow at inference, and storage-intensive, motivating more efficient adaptation. The paper introduces AdapterDrop and AdapterFusion pruning, reporting faster multi-task inference while largely or entirely preserving task performance.
Problem
Transformer models require millions or billions of parameters, resulting in slow inference and large storage requirements.
Method
The paper removes adapters from lower transformer layers and prunes least-activated adapters from AdapterFusion models.
Results
60% faster training is reported for adapters than full model fine-tuning, while AdapterDrop improves multi-task inference speed and AdapterFusion pruning maintains task performance entirely.
Takeaways & Limitations
AdapterDrop dynamically reduces computational overhead for multi-task inference, while AdapterFusion pruning improves efficiency without reducing task performance.
Takeaways & Limitations
The authors identify more efficient pretrained adapters, cross-layer weight sharing, and training-time AdapterFusion pruning as future directions.
Abstract
from arXiv · showhide
Massively pre-trained transformer models are computationally expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.
1 Introduction
Transformer models are costly to fine-tune, slow at inference, and storage-intensive. The paper addresses adapter efficiency by introducing dynamically removable adapters and evaluating their computational benefits.
- Transformer-based models require millions or billions of parameters, resulting in slow inference and large storage requirements.
- Adapters train newly introduced weights at each layer while sharing most parameters between tasks.
- The paper establishes computational efficiency for two adapter architectures during training and inference.
- AdapterDrop dynamically removes adapters from lower transformer layers during training and inference, adjusting models to available computational resources.
- 60% faster training and 4–6% slower inference are reported for adapters relative to full model fine-tuning.
- 39% faster inference is achieved with adapters dropped from the first five layers when performing inference on 8 tasks simultaneously.
2 Efficiency of Adapters
Adapters reduce training overhead because most model parameters remain frozen, although their additional components make inference slightly slower than full fine-tuning.
- Adapters add forward-pass components but avoid backpropagation through the entire model during fine-tuning.
- 60% faster training is observed for adapters in some configurations compared with full model fine-tuning.
- The two adapter architectures differ only marginally in training efficiency, with Pfeiffer adapters slightly faster because of their simpler architecture.
- Adapter training speedups arise from decreased gradient-computation overhead because most parameters are frozen.
- 94–96% inference speed relative to fully fine-tuned models is achieved by both adapter architectures, depending on input size.
3 AdapterDrop
AdapterDrop removes adapters from lower transformer layers to improve efficiency while preserving much of task performance. Robust AdapterDrop additionally supports dynamically varying the number of dropped layers.
- AdapterDrop dynamically removes adapters from lower transformer layers, where lower layers often have a small impact on task performance.
- Specialized AdapterDrop removes adapters from the first n layers with n fixed during training, producing separate models for each n.
- Robust AdapterDrop samples n from [0, 11] for each training batch, producing one model applicable to varying numbers of dropped layers.
- 97.1% of original performance is maintained by specialized AdapterDrop with the first five layers dropped, averaged over eight GLUE tasks.
- 95.4% of original performance is maintained by robust AdapterDrop with five layers dropped, on average.
- 21–42% inference speedups are measured for multiple simultaneous tasks with five layers dropped.
- Randomly dropping adapters from selected layers generally performs worse and requires choosing a suitable dropout rate.
4 Efficiency of AdapterFusion
AdapterFusion combines task-specific adapter representations but is computationally expensive because all included adapters are processed sequentially.
- AdapterFusion learns an optimal combination of output representations from adapters trained for different tasks.
- Sequentially processing all included adapters makes AdapterFusion computationally expensive.
- 47% slower training and 62% slower inference are observed for AdapterFusion with eight adapters compared with a fully fine-tuned model.
5 AdapterDrop for AdapterFusion
The paper improves AdapterFusion efficiency by removing entire early fusion layers or pruning low-importance adapters, targeting substantial inference costs while preserving task performance where possible.
- Approach: The authors evaluate two efficiency strategies for AdapterFusion: removing entire AF layers and pruning the least important adapters.The strategies address AF efficiency primarily at inference time.
- Removing AdapterFusion layers: On RTE, AdapterFusion remains better than the single-task adapter when AF is removed from the first five layers, improving inference efficiency by 26%.The evaluation progressively removes AF from lower layers at test time.
- Removing AdapterFusion layers: On CoLA, removing AF from the first layer causes more noticeable performance decreases, illustrating a performance-efficiency tradeoff across tasks.The paper notes that AdapterDrop may not suit all tasks and that CoLA is an extreme case.
- AdapterFusion pruning: AdapterFusion pruning ranks adapters by average activation and removes those with the lowest activations from each fusion layer.The ranking is computed using instances from the respective AdapterFusion training set.
- AdapterFusion pruning: With two adapters remaining, pruning achieves comparable performance to eight-adapter Fusion models while improving inference speed by 68%.The authors recommend pruning before deploying AdapterFusion models.
6 Conclusion
Adapters provide faster training than full fine-tuning, while AdapterDrop and AdapterFusion pruning reduce inference overhead with limited or no performance loss. The paper also identifies several directions for extending efficient adapter-based models.
- Up to 60% faster training steps come with 4–6% slower inference than full model fine-tuning.The authors conclude adapters are advantageous when training efficiency is more important.
- AdapterDrop dynamically removes adapters from lower transformer layers to reduce multi-task inference overhead while largely maintaining task performance.The approach supports models making multiple independent predictions on one input.
- Pruning the least activated adapters in AdapterFusion improves efficiency while maintaining performance entirely.Dropping entire fusion layers instead creates a considerable performance–efficiency tradeoff.
- Future work includes more efficient pretrained adapters, sharing adapter weights across layers, and pruning AdapterFusion adapters during training.Preliminary results for related ideas are presented in the appendix as starting points for future work.
A.2 Task Performances
The task-performance experiments use RoBERTa-base on eight GLUE tasks to compare adapter convergence with standard fine-tuning and to test masked-language-model initialization. Adapters show similar convergence, while MLM initialization provides no substantial benefit.
- The experiments evaluate downstream performance on GLUE using RoBERTa-base as the pretrained model.Larger datasets are trained for 10 epochs and the remaining datasets for 20 epochs.
- The study measures both convergence behavior and whether masked-language-model initialization leads to faster convergence.These are the two stated experimental questions.
- Adapters show similar convergence behavior to standard RoBERTa-base on several of the eight GLUE tasks.The study excludes WNLI and reports mean and standard deviation over three runs.
- Across all tasks, MLM-pretrained adapter weights do not substantially affect downstream task convergence compared with random initialization.The experiments find no evidence that the MLM objective improves convergence performance.
C Detailed Results: AdapterDrop Task Performances
This section evaluates task performance under AdapterDrop and explores cross-layer adapter weight sharing. Shared weights largely preserve performance, and smaller compression rates reduce the remaining gap while using fewer parameters.
- AdapterDrop Task Performances: The section reports detailed AdapterDrop task performances and relative differences from a standard adapter without AdapterDrop.These comparisons are presented for different training strategies.
- Adapter with Cross-Layer Parameter Sharing: The experiments use the Pfeiffer adapter architecture and vary its bottleneck down-projection compression rate to study capacity effects.The standard compression rate is 16, while smaller values produce larger model capacity.
- Adapter with Cross-Layer Parameter Sharing: Cross-layer parameter sharing with compression rate 16 largely maintains performance, with an average difference of 2.35% versus separate weights.The shared adapter with compression rate 16 requires only 307KB of storage.
- Adapter with Cross-Layer Parameter Sharing: A compression rate of 4 closes the performance gap by more than 50% while requiring 66% fewer parameters.Smaller compression rates increase adapter model capacity.
E Training AdapterFusion with Dropout
AdapterFusion dropout can substantially accelerate training, but its task-performance effects vary by downstream task. Inference efficiency can also be improved by reducing fused adapters, with AdapterDrop helping preserve performance when fusion layers are omitted.
- Training AdapterFusion with Dropout: 74% faster training steps result from dropping 6 of 8 adapters with a 75% AdapterFusion dropout rate.This average uses sequence length 128 and batch size 32 on the four smallest GLUE tasks.
- Training AdapterFusion with Dropout: Fusion dropout decreases performance on RTE and CoLA, has little impact on STS-B, and improves MRPC.There is no difference on STS-B when dropping out 25% of adapters.
- Training AdapterFusion with Dropout: Fusion-dropout effectiveness depends on the individual downstream task and may suit resource-constrained settings.The authors do not report a uniform task-performance trend.
- AdapterFusion Inference: Dropping AdapterFusion layers during inference is evaluated on MRPC, STS-B, CoLA, and RTE.The comparison includes standard adapters and adapters created with AdapterDrop.
- AdapterFusion Inference: Using AdapterDrop when creating adapters or applying it to AdapterFusion significantly reduces performance loss when fusion layers are omitted.The strongest performance drop occurs without AdapterDrop in the adapters or fusion.
G.2 AdapterDrop
AdapterDrop improves efficiency during multi-task inference and training by dropping adapters from lower layers, with speed gains also examined for AdapterFusion.
- Multi-task inference: AdapterDrop provides speedups for multi-task inference compared with sequentially processed fully fine-tuned models.The evaluation uses simultaneous task processing and reports speedup for each additional dropped layer.
- Training adapters with dropped layers: 4.7% average training speedup per dropped layer is reported on V100, compared with 4.5% on TitanX.The averages cover batch sizes 16, 32, and 64 and sequence lengths 64, 128, and 256.
- AdapterFusion: AdapterFusion efficiency is evaluated as a function of the number of included adapters and dropped layers during training and inference.The comparisons include fully fine-tuned models and models with one adapter, while measuring computational overhead from additional composed adapters.
H Parallel Implementation of AdapterFusion
The parallel AdapterFusion implementation reformulates adapter linear layers as convolutions, but its efficiency depends on input size and GPU utilization.
- Parallel implementation: Parallel AdapterFusion computes different adapters concurrently by reformulating their linear layers as two convolutions.The first convolution produces adapter-specific downprojection channels, while the second is grouped and returns adapter-specific hidden-dimension channels.
- Parallel implementation: The first convolution uses hidden-dimension kernel size and adapter-count-scaled downprojection output channels.The second grouped convolution processes downprojection-sized channel blocks and outputs adapter-count-scaled hidden-dimension channels.
- Efficiency comparison: The iterative implementation is faster for larger inputs, whereas parallel processing is faster for sufficiently small inputs when the GPU is under capacity.The comparison is reported across batch sizes, sequence lengths, adapter counts, and two GPUs.
- Efficiency comparison: Inference and training time are also compared for adapter architectures, fully fine-tuned models, and models without adapters.Measurements include batch-size and sequence-length effects, including GPU under-capacity behavior for small inputs.
- Efficiency comparison: AdapterFusion speed and overhead are measured across adapter counts, sequence lengths, batch sizes, and dropped layers during training and inference.The supplied figures and tables compare AdapterFusion with fully fine-tuned and single-adapter models, and report AdapterDrop speedups.
- Initialization: Adapter initialization strategies are compared for RoBERTa-base after 10k training steps under specified learning rates.The comparison covers random initialization and masked-language-model pre-training.
- Task performance: AdapterDrop performance is evaluated across eight GLUE tasks against standard adapters and separately specialized AdapterDrop models.The evaluation varies the number of dropped layers and reports retained task performance relative to the standard adapter.