AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

1 Introduction

• Full fine-tuning updates all PLM parameters, while PEFT reduces trainable parameters but typically underperforms full tuning, leaving a performance gap.The motivation includes the cost of storing a full model copy for each task as PLMs grow.Source: §1 · p. 1
• AdaMix replaces a single adaptation module with multiple modules that learn task views, routing each input to one module to match single-module FLOPs.The approach is described for adapters and LoRA low-rank decompositions.Source: §1 · p. 2
• Stochastic routing avoids extra module-selection parameters, while consistency regularization and shared modules address instability from random routing.Merging combines adaptation weights into one module per layer, retaining the single-module adaptation-parameter count.Source: §1 · p. 2
• Tuning only 0.1 −0.2% of a pre-trained language model’s parameters, AdaMix is the first PEFT method to outperform full model fine-tuning on all NLU tasks on GLUE, and competing methods for NLG and few-shot NLU tasks.Source: §1 · p. 2
• AdaMix reduces per-task storage by up to 444x in the RoBERTa-large setting, from 355MB to 0.8MB.PEFT also lets tasks share one frozen PLM copy.Source: §1 · p. 2

2.1 Mixture-of-Experts

• MoE uses N feed-forward experts and a gate to route each token through a few experts, supporting conditional computation while keeping per-example FLOPs constant.Routing to one expert per token can match the FLOPs of a dense Transformer.Source: §2.1 · p. 3
• The gate’s output gives the probability of selecting each expert, and the sparse MoE layer combines expert outputs according to the gate.When N = 1, the sparse model reduces to a traditional dense model.Source: §2.1 · p. 3

2.2 Adapters

• Adapters update only their added parameters while the PLM stays frozen; each adapter down-projects, applies a nonlinearity, then up-projects within residual connections.The down projection maps from model dimension d to bottleneck dimension r, and the up projection maps back to d.Source: §2.2 · pp. 3–4

3 Mixture-of-Adaptations

• AdaMix extends adapters and other PEFT mechanisms by placing multiple adaptation modules in each Transformer layer instead of a single module.Adapters are the running example; the paper also names LoRA as an underlying PEFT mechanism.Source: §3 · p. 4

3.1 Routing Policy

• At each training step, stochastic routing selects an up/down projection pair per layer, and every example in the batch uses that same pair.Random routing avoids additional expert-selection parameters or computation and does not require load balancing.Source: §3.1 · p. 4

3.2 Consistency regularization

• Consistency regularization compares predictions from two stochastic module sets to encourage information sharing and prevent divergence.The PLM remains frozen while only adaptation-module parameters are updated.Source: §3.2 · pp. 4–5

3.3 Adaptation module merging

• At inference, averaging corresponding adaptation weights collapses the alternatives to one module per layer, reducing serving cost.The paper motivates averaging as a way to avoid retaining several adaptation modules.Source: §3.3 · p. 5

3.4 Adaptation module sharing

• Sharing adaptation modules addresses the increased tuning burden of multiple module sets when labeled data are limited; the standard adapter setting shares projection-up matrices.The paper investigates sharing choices in its ablations.Source: §3.4 · p. 5

3.5 Connection to Bayesian Neural Networks and Model Ensembling

• AdaMix frames random routing as stochastic regularization and a tractable surrogate for the model posterior, then merges weights to avoid storing sampled models at inference.The paper contrasts this with logit ensembling, which requires retaining copies of all adaptation weights.Source: §3.5 · pp. 5–6

4.1 Experimental Setup

• Experiments cover eight GLUE NLU tasks and E2E, WebNLG, and DART generation, comparing AdaMix with full fine-tuning and state-of-the-art PEFT baselines.NLU uses BERT-base and RoBERTa-large; NLG uses GPT-2.Source: §4.1 · p. 6

4.2.1 NLU Tasks

• AdaMix with adapters is the only PEFT method reported to outperform full model fine-tuning on every NLU task and on average.The GLUE results use RoBERTa-large and BERT-base encoders.Source: §4.2.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.2.2 NLG Tasks

• On E2E, DART, and WebNLG, AdaMix outperforms competing methods and full fine-tuning; adapters and LoRA are evaluated as underlying PEFT methods.For E2E, AdaMix with each method also outperforms its corresponding underlying PEFT method.Source: §4.2.2 · p. 7, Table 3 · p. 8, Table 4 · p. 8

4.2.3 Few-shot NLU

• At |K| = 30 with a frozen RoBERTa-large encoder, AdaMix with adapters outperforms full model tuning in few-shot NLU.The comparison covers six GLUE tasks.Source: §4.2.3 · p. 7

4.3 Ablation Study

• AdaMix with adaptation merging performs better than non-merging routing variants; Figure 5 shows merged performance above the random-routing average and comparable to its best performance.The ablation compares random routing and fixed routing without merging.Source: §4.3 · pp. 7–8, Figure 5 · p. 9
• Weight averaging outperforms logit ensembling, which requires four random-routing passes and costs 4 × AdaMix inference time.The reported averaging comparison uses three seeds.Source: §4.3 · p. 8
• Removing consistency regularization significantly degrades performance, while removing module sharing widens the performance gap on smaller datasets and slows convergence.The sharing comparison reports faster convergence and lower training loss with sharing for the same number of training steps.Source: §4.3 · pp. 8–9
• Increasing the number of adaptation modules yields diminishing aggregate returns and can hurt low-resource tasks; increasing adapter bottleneck dimension improves performance with diminishing returns.The module-count comparison varies 2, 4, and 8 modules.Source: §4.3 · p. 9

Related Work

• PEFT approaches tune a subset of existing parameters or introduce new components, including adapters, prompts, prefixes, and low-rank adaptations.AdaMix is presented as a mixture of adaptation modules applicable to PEFT methods such as adapters and low-rank decompositions.Source: §5 · pp. 9–10
• Unlike prior sparse MoE work described here, which pre-trains the entire model, AdaMix adapts pretrained language models by tuning a small number of sparse adapter parameters.AdaMix uses stochastic routing during training and merging during inference to retain the single-module computational cost.Source: §5 · p. 10
• Where prior weight-averaging work focuses on full-model weights, AdaMix averages the small adaptation-module weights while keeping the large model parameters fixed.The cited prior work includes averaging independently fine-tuned models and models tuned with different hyperparameter configurations.Source: §5 · p. 10

• The conclusion presents AdaMix as a framework for PEFT of large pre-trained language models.Source: §6 · p. 10
• Across NLU and NLG tasks, AdaMix improves downstream performance with adapters and low-rank decompositions without increasing the underlying method’s computational cost.Source: §6 · p. 10
• AdaMix outperforms full model fine-tuning and other state-of-the-art PEFT methods while tuning only 0.1–0.2% of PLM parameters.Source: §6 · p. 10

7 Limitations

• AdaMix training is more compute-intensive than standard PEFT, typically requiring 1–2 times as many training iterations and negatively affecting carbon footprint.Source: §7 · p. 10
• Experiments cover adapters and LoRA, while combinations with prompt-tuning and prefix-tuning remain untested.Source: §7 · p. 10

A Few-shot NLU Datasets

• Few-shot experiments cover six tasks: MNLI, RTE, QQP, SST-2, MPQA, and Subj.Source: A Few-shot NLU Datasets · p. 14
• The setup uses fixed training epochs without validation data for hyperparameter tuning or early stopping, and reports means and standard deviations across five seeds.Source: A Few-shot NLU Datasets · p. 14
• AdaMix is trained with prompt-based fine-tuning, using the same few-shot labeled instances as prior work and no unlabeled data.Source: A Few-shot NLU Datasets · p. 14

B Ablation Study

• The ablation study examines the impact of parameter sharing in the AdaMix adapter framework.Source: Table 11 · p. 14

C Detailed Results on NLU Tasks

• AdaMix with RoBERTa-large achieves the best performance across GLUE task metrics, and AdaMix with adapters is the only PEFT method reported to beat full fine-tuning on every task and on average.The comparison includes results with BERT-base and RoBERTa-large encoders.Source: C Detailed Results on NLU Tasks · p. 14
• Table 12 varies adapter bottleneck dimensions in AdaMix with BERT-base and RoBERTa-large; an asterisk marks the dimension used in AdaMix with adapters.Source: Table 12 · p. 14
• 2.2% improvement over full model fine-tuning and 1.2% over UNIPELT are reported for AdaMix with BERT-base; the improvement is consistent across tasks with RoBERTa-large.Source: C Detailed Results on NLU Tasks · p. 14

D Hyper-parameter

• Table 13 covers GLUE development-set results with BERT-base, reports task bests, and defines its average using F1 for QQP and MRPC.Its parameter-count column refers to updated parameters at inference.Source: Table 13 · p. 15
• Figure 6 plots AdaMix-RandomRouting performance distributions for BERT-base and RoBERTa-large, with a red dot marking AdaMix performance.Source: Figure 6 · p. 15
• Figure 7 presents a convergence analysis of the adapter-sharing design in AdaMix.Source: Figure 7 · p. 15
• Table 14 compares routing and ensembling strategies with AdaMix on the GLUE development set using BERT-base and RoBERTa-large encoders; averages follow Tables 1 and 2 for consistency.Source: Table 14 · p. 16
• Hyperparameter configurations are provided for GLUE tasks and GPT-2 Medium NLG tasks; the latter retains other default LoRA training and generation settings.Source: D Hyper-parameter · p. 14, Table 15 · p. 16, Table 16 · p. 17