PaperlayerView exact revision on arXiv

Source-linked AI summary

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder

arXiv:2106.04647v2cs.CL

TL;DR

Fine-tuning large pretrained language models is costly and can be unstable in low-resource settings, while existing parameter-efficient methods trade compactness against performance. Compacter combines adapters, low-rank optimization, and hypercomplex multiplication through Kronecker-factorized task-specific weights. It trains only 0.047% of parameters while matching or exceeding full fine-tuning across the reported benchmarks and low-resource settings.

  • Problem

    Full fine-tuning requires updating and storing all model parameters, while parameter-efficient alternatives can use many parameters or underperform standard fine-tuning.

  • Method

    Compacter inserts task-specific weight matrices computed as sums of Kronecker products between shared slow weights and layer-specific fast rank-one matrices.

  • Results

    0.047% of a pretrained model’s parameters are trained, with performance on par with or better than full fine-tuning on GLUE, SuperGLUE, and low-resource settings.

  • Takeaways & Limitations

    Compacter offers a lightweight adaptation method with substantially fewer trainable parameters while retaining strong full-data performance and improving results in data-limited scenarios.

  • Takeaways & Limitations

    Some competing low-parameter methods incur substantial memory overhead or slow training, limiting their application to large-scale pretrained language models.

Abstract

from arXiv · show

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~\url{https://github.com/rabeehk/compacter}.

1 Introduction

Large pretrained language models are costly to adapt because full fine-tuning updates and stores every parameter, while existing parameter-efficient methods trade parameter count against performance. Compacter targets a better balance by combining adapter, low-rank, and hypercomplex ideas.

  • Motivation: Full fine-tuning updates all parameters and requires a separate model copy per task, creating storage and deployment costs.It can also be unstable and perform poorly on low-resource datasets.
  • Prior approaches: Parameter-efficient methods keep pretrained parameters fixed but trade trainable-parameter count against task performance.Prompt-based methods may require huge models, while low-rank methods incur projection-matrix memory and training-time overhead.
  • Compacter: Compacter combines adapters, low-rank methods, and hypercomplex multiplication layers to target task performance, trainable parameters, and memory footprint jointly.It inserts task-specific weight matrices computed from shared slow weights and layer-specific fast rank-one matrices.
  • Results: 0.047% of a pretrained model’s parameters are trained by Compacter in practice.Its adapter parameter complexity is O(k+d), compared with O(kd) for regular adapters.
  • Results: On GLUE and SuperGLUE, Compacter outperforms other parameter-efficient methods and matches or exceeds full fine-tuning.The supplied introduction reports stronger performance than standard fine-tuning in low-resource settings.

2 Background

Adapter layers provide a parameter-efficient way to adapt pretrained models by inserting small trainable modules while keeping the original model fixed. They use bottleneck down- and up-projections around a nonlinearity within transformer layers.

  • Kronecker products: The Kronecker product combines matrices A∈R^m×f and B∈R^p×q into A⊗B∈R^mp×fq.Its block structure places scaled copies of B according to the elements of A.
  • Adapter layers: Adapter methods insert small task-specific modules into pretrained models and train them with layer normalizations while keeping the remaining pretrained parameters fixed.This allows models to adapt to new tasks without updating the full pretrained network.
  • Adapter placement: Transformer adapters are inserted after both attention and feed-forward blocks, before their skip connections.Each transformer layer contains these two primary modules, each followed by a skip connection.
  • Adapter architecture: Adapters are bottleneck architectures whose output dimension remains similar to the input, preserving the original model’s structure and parameters.The adapter uses a down-projection, GeLU nonlinearity, and up-projection.

3 Method

COMPACTER replaces standard adapter projections with low-rank parameterized hypercomplex multiplication layers. It shares slow matrices across adapter layers while learning fast, layer-specific rank-one factors, reducing complexity to O(k+d).

  • Hypercomplex weight construction: Each weight matrix is computed as a sum of Kronecker products between shared matrices Ai and adapter-specific matrices Bi.The construction assumes k and d are divisible by n and uses n Kronecker products.
  • Sharing information across adapters: COMPACTER shares Ai across adapter layers while Bi captures information specific to each individual layer.The shared Ai act as slow weights, whereas the adapter-specific Bi act as fast weights.
  • Low-rank parameterization: Bi is generally parameterized as a rank-one matrix by multiplying two independent rank-one weights, si and ti.The rank can be increased for more complex target tasks, but the general setting uses r=1.
  • Efficiency: COMPACTER reduces adapter-layer parameter complexity to O(k+d), compared with O(kd) for regular adapters.The reduction follows from combining shared matrices with low-rank adapter-specific factors.
  • COMPACTER: COMPACTER replaces adapter down- and up-projection layers with low-rank parameterized hypercomplex multiplication layers.These layers are introduced as an efficient extension of adapter layers using parameterized hypercomplex multiplication.

4 Parameter Efficiency

COMPACTER substantially reduces trainable parameters by sharing hypercomplex components across layers and using rank-one adapter-specific factors. Under the stated condition, its complexity is O(k+d), and in T5BASE it trains only 0.047% of parameters while maintaining comparable full-fine-tuning performance.

  • Adapters parameters: Regular adapters use 2L(2kd) parameters for a transformer with L encoder and decoder layers.Each layer contains two adapters, with down- and up-projection matrices containing kd parameters each.
  • PHM-ADAPTER parameters: PHM-ADAPTER has complexity O(kd/n) when kd>n^4, reducing parameters by almost 1/n relative to standard fully connected layers.This scaling is stated for typical adapter and large-language-model settings.
  • COMPACTER parameters: COMPACTER has 4L(k+d)+n^3 parameters for a transformer with L encoder and decoder layers.The n^3 shared-matrix term remains constant across layers, while the layer-dependent term scales with k+d.
  • COMPACTER parameters: When 4L(k+d)>n^3, COMPACTER has O(k+d) complexity, compared with O(kd) for adapters and O(kd/n) for PHM-ADAPTER.The comparison applies in settings with a large number of layers under the stated mild condition.
  • Empirical parameter efficiency: 0.047% of T5BASE’s 222M parameters are learned by COMPACTER while maintaining comparable performance to full fine-tuning.Sharing the matrices Ai across layers prevents their parameter count from scaling linearly with L.

5 Experiments

Experiments compare Compacter and related parameter-efficient methods with full fine-tuning across GLUE, SuperGLUE, efficiency, and low-resource settings. Compacter methods achieve strong benchmark performance while using fewer trainable parameters, with trade-offs in memory and training speed.

  • Results on GLUE: Compacter and Compacter++ outperform previous parameter-efficient methods on GLUE while performing on par with full fine-tuning using 0.07% and 0.047% of parameters.These results are reported for T5BASE.
  • Results on GLUE: INTRINSIC-SAID trains only 0.009% of parameters but averages 1.05 points below full fine-tuning, while random projections create memory and training-time costs.The method stores random projection matrices and is slow to train.
  • Results on SuperGLUE: Compacter and Compacter++ outperform other parameter-efficient methods and full fine-tuning on SuperGLUE while training 0.073% and 0.048% of parameters.The authors observe a pattern similar to GLUE.
  • Efficiency Evaluation: 0.047% of parameters gives Compacter++ the best combination of average GLUE performance and parameter efficiency among the compared methods.Compacter++ also reduces memory usage by 41.94% and training time by 26.51% relative to T5BASE.
  • Efficiency Evaluation: PROMPT TUNING and INTRINSIC-SAID use the fewest parameters but have high memory overhead, slow training, and performance substantially below full fine-tuning.Prompt tuning is sensitive to initialization and learning rate, especially on low-resource datasets.
  • Low-resource Fine-tuning: Compacter++ substantially improves GLUE results in low-resource settings, indicating more effective fine-tuning in this regime.The evaluation subsamples each task to 100, 500, 1000, 2000, or 4000 training samples.

6 Related Work

Related work covers adapter-based, low-dimensional, sparsification, and hypercomplex approaches to parameter-efficient model adaptation. Hypercomplex methods reduce parameter degrees of freedom, but Compacter is presented as the first effort to apply them to efficient fine-tuning of large-scale language models.

  • Hypercomplex representations: Hypercomplex representations replace standard matrix multiplications with Hamilton products, offering up to a 4× parameter saving in a single multiplication operation.
  • Hypercomplex representations: Hypercomplex layers can reduce fully connected-layer parameters to 1/n under a mild condition, but no prior work had leveraged them for efficient language-model fine-tuning.
  • Low-rank methods optimize in a lower-dimensional randomly oriented subspace, while sparsification methods reduce parameters but remain poorly supported by current hardware.

7 Conclusion

The conclusion presents Compacter as a lightweight fine-tuning method that combines shared slow weights with layer-specific fast rank-one matrices. It substantially reduces adapter parameters while matching or exceeding standard fine-tuning, especially in data-limited settings.

  • Compacter generates weights by summing Kronecker products between shared slow weights and layer-specific fast rank-one matrices.
  • O(kd) adapter parameters are reduced to O(k+d) through Compacter’s weight-generation formulation.
  • 2127.66× fewer parameters than standard fine-tuning still yields comparable or better full-data performance and better performance in data-limited scenarios.

A Experimental Details

The experiments use GLUE and SuperGLUE datasets cast into sequence-to-sequence format, with task-specific preprocessing, hardware, sequence lengths, batch sizes, and learning-rate settings. Results tables report benchmark performance and parameter counts across methods.

  • Datasets: Experiments cover the standard GLUE and SuperGLUE benchmarks, including classification, similarity, entailment, reasoning, and reading-comprehension tasks.
  • Data preprocessing: STS-B is recast as a 21-class classification task by rounding target scores to the nearest increment of 0.2.
  • Computing infrastructure: Experiments use one NVIDIA GeForce RTX 3090, while low-resource experiments use one GeForce GTX 1080 Ti GPU.
  • Training hyper-parameters: GLUE uses maximum sequence length 128 and batch size 100, while SuperGLUE uses length 256 and batch size 32, except ReCoRD at length 512 and batch size 16.
  • Reported measurements: Table 6 reports GLUE performance under different hyper-parameters, including total and per-task trainable parameters relative to T5BASE.

B Impact of Hyper-parameters

The hyper-parameter analysis examines dimension and n across intrinsic, PHM-adapter, and Compacter variants. It finds task-dependent sensitivity for INTRINSIC-SAID, substantial parameter reductions for PHM-ADAPTER and Compacter, and near-constant Compacter parameter counts across n.

  • Impact of dimension (d′) on INTRINSIC-SAID: Increasing INTRINSIC-SAID dimension d′ often improves results, but d′ must be tuned for each dataset to achieve optimal performance.
  • Impact of n on PHM-ADAPTER: PHM-ADAPTER is evaluated with n = {4,8,12} and bottleneck sizes d∈{24,48,96}.
  • Impact of n on PHM-ADAPTER: For T5BASE, the condition kd > n4 is partially or fully satisfied depending on d and n, and it is satisfied for the listed larger T5 models and settings.
  • Impact of n on PHM-ADAPTER: 5.2× fewer parameters are obtained by PHM-ADAPTER with n=8 compared to ADAPTER.
  • Impact of n on COMPACTER: Compacter trainable-parameter counts remain almost constant across n because layer-normalization and bias parameters comprise a large share.
  • 11.4× fewer parameters are used by COMPACTER than ADAPTER, while COMPACTER++ reduces parameters by 17.7× in total.

C Results with Fine-tuning the Output Layer

Fine-tuning the output layer reduces relative parameter savings because the output layer dominates each method’s parameters, while Compacter remains effective with or without this tuning.

  • Output-layer parameters dominate each method, reducing the relative parameter savings reported in Table 7.
  • The standard adapter shows the largest performance improvement from fine-tuning the output layer compared with Table 1.
  • Compacter’s proposed methods perform well both with and without fine-tuning the output layer.

D Results on SUPERGLUE

The paper evaluates its proposed methods on SUPERGLUE across different values of n, reporting parameter counts and task-specific performance metrics.

  • SUPERGLUE performance is evaluated for the proposed methods across different values of n.
  • Table 8 reports total parameters across tasks and each task’s trained-parameter percentage relative to T5BASE.
  • The reported SUPERGLUE metrics include accuracy, F1, exact match, and task-specific scores across the benchmark’s tasks.

E Impact of Model Size

With T5SMALL, all parameter-efficient methods perform worse than full fine-tuning, unlike some results with the larger T5BASE model; adapters and the proposed methods perform best among them.

  • 60M-parameter T5SMALL experiments evaluate adapter bottleneck sizes {16,32,64} and proposed-method values n={4,8,16}.
  • All parameter-efficient fine-tuning methods perform worse than full fine-tuning at the T5SMALL model size.
  • With 222M-parameter T5BASE, some parameter-efficient methods perform on par with or outperform full fine-tuning.
  • Among the T5SMALL methods, adapters and the proposed methods perform best.
  • Tables report total parameters and trained-parameter proportions relative to the corresponding T5SMALL or T5BASE model.
Loading 2106.04647v2…