PaperlayerView exact revision on arXiv

Source-linked AI summary

Composing Parameter-Efficient Modules with Arithmetic Operations

Jinghan Zhang, Shiqi Chen, Junteng Liu, Junxian He

arXiv:2306.14870v2cs.CL

TL;DR

The paper addresses how to combine the diverse capabilities stored in parameter-efficient modules without retraining or accessing task data. It defines arithmetic operators over module parameters, applies them across several composition settings, and reports successful composition with significant gains across those settings. The approach is also extended to detoxifying Alpaca-LoRA, while the authors note constraints on architecture, initialization, and hyperparameter selection.

  • Problem

    PEFT produces separate compact modules for different capabilities, creating a need to compose them flexibly while the underlying pretrained model remains unchanged.

  • Method

    The paper defines addition and negation operators for parameter-efficient modules and combines them through linear arithmetic in parameter space without additional training.

  • Results

    The approach successfully composes PEMs without additional training and achieves significant gains across distribution generalization, multi-tasking, unlearning, and domain transfer settings.

  • Takeaways & Limitations

    Arithmetic composition yields new parameter-efficient modules for combining, removing, and transferring capabilities, including detoxification of Alpaca-LoRA.

  • Takeaways & Limitations

    The study is restricted mainly to identical PEM architectures and shared initialization, and requires tuning the weight hyperparameter λ.

Abstract

from arXiv · show

As an efficient alternative to conventional full finetuning, parameter-efficient finetuning (PEFT) is becoming the prevailing method to adapt pretrained language models. In PEFT, a lightweight module is learned on each dataset while the underlying pretrained language model remains unchanged, resulting in multiple compact modules representing diverse skills when applied to various domains and tasks. In this paper, we propose to compose these parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities. Specifically, we first define addition and negation operators for the module, and then further compose these two basic operators to perform flexible arithmetic. Our approach requires \emph{no additional training} and enables highly flexible module composition. We apply different arithmetic operations to compose the parameter-efficient modules for (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. Additionally, we extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. Empirical results demonstrate that our approach produces new and effective parameter-efficient modules that significantly outperform existing ones across all settings.

1 Introduction

The paper asks whether compact parameter-efficient modules can be composed to combine or manipulate capabilities without retraining. It proposes linear arithmetic composition and evaluates it across generalization, multi-tasking, unlearning, domain transfer, and instruction-tuned models.

  • Motivation: PEFT trains small modules while keeping most pretrained parameters frozen, producing compact modules associated with distinct capabilities.These modules can be transferred with low memory and storage cost.
  • Approach: The paper composes trained modules in parameter space to integrate, remove, or transfer capabilities without accessing annotated data.The training-free objective addresses restricted access caused by data privacy and intellectual-property concerns.
  • Approach: Addition aggregates module skills, while negation retracts selected abilities from the underlying pretrained model.The operators are defined for the studied parameter-efficient module architecture.
  • Evaluation: The experiments cover distribution generalization, multi-tasking, unlearning, and domain transfer using LoRA and (IA)3 modules.The study also extends composition to detoxify Alpaca-LoRA.
  • Results: The proposed approach composes PEMs without additional training and reports significant gains from arithmetic combinations of existing modules.The reported result spans all evaluated settings.

2 Background

PEFT adapts pretrained language models by updating lightweight modules while leaving most pretrained parameters unchanged. The paper focuses on LoRA and (IA)3 as compact module architectures for composition.

  • PEFT background: Adapters introduced PEFT by inserting small modules into pretrained transformers and updating only adapter parameters during finetuning.The adapter uses a bottleneck projection from the input dimension to a lower-dimensional space and back.
  • Studied architectures: LoRA and (IA)3 are identified as recent state-of-the-art PEFT methods and the architectures studied in this paper.Both methods form parameter-efficient modules intended for later composition.
  • LoRA: LoRA applies low-rank projection matrices A and B to selected transformer weight matrices, with rank r much smaller than the matrix dimensions.The study follows the established practice of applying LoRA to query and value projections in attention.
  • LoRA: In LoRA, A is randomly initialized and B is initialized to zeros so the pretrained model is recovered at tuning start.The LoRA module is represented by θ_lora = {A, B}.
  • (IA)3: (IA)3 introduces trainable vectors that rescale attention keys, attention values, and inner activations in feed-forward networks.Its vectors are initialized to ones so the model is unchanged at the beginning of tuning.

3 Composition through Arithmetic Operation

The method composes parameter-efficient modules by defining addition and negation in parameter space, then combining them with weighted arithmetic for several objectives. Negation is designed around the hidden-state modification produced by a module rather than simple parameter sign reversal.

  • Motivation: Prior work motivates arithmetic composition from parameter merging after a shared pretrained initialization, while this paper adapts the idea to PEMs.The paper contrasts its training-free arithmetic with prior output-fusion approaches.
  • Basic operators: PEM addition pairs corresponding parameters and adds them component-wise to create a module containing collective features.The operator applies to both LoRA and (IA)3 parameters.
  • Basic operators: Negation targets the module’s hidden-state modification Δh, enabling subtraction of abilities such as skills learned from toxic data.Simple negation of all LoRA parameters would not change the output, so the operator is formulated around Δh.
  • Basic operators: For LoRA, the study negates B while leaving A unchanged; for (IA)3, it derives l_neg = 2 − l for each scaling vector.The two architectures therefore require different parameter transformations to negate their hidden-state effects.
  • Composed operations: Weighted composition uses λ to interpolate added modules or scale negation, with λ tuned on a validation set.The paper studies addition for generalization and multi-tasking, negation for unlearning, and subtraction-plus-addition for domain transfer.

4 Experiments

The experiments evaluate arithmetic composition of parameter-efficient modules across distribution generalization, multi-tasking, unlearning, domain transfer, and instruction-tuned language models. Across these settings, composition is performed without additional training and yields improvements, while task trade-offs and initialization effects are also examined.

  • Experimental setup: Experiments fix pretrained checkpoints and PEM architectures within each scenario, evaluating LoRA and (IA)3 alongside full-finetuning reference results.The full-finetuning results are not directly comparable because the paper focuses on composing parameter-efficient modules.
  • Distribution generalization: The merged PEM combines modules trained on divergent distributions through θmerge = λθ(1) + (1 − λ)θ(2) and is evaluated on the original validation distribution.This setup tests whether combining subset-specific modules improves generalization to the union of their distributions.
  • Distribution generalization: 5.2 and 4.0 absolute points are the RTE gains for merged LoRA and (IA)3 modules, respectively, over the average performance of their individual modules.The results indicate that addition can integrate abilities learned from divergent distributions.
  • Multi-tasking: Merged PEMs show minor drops on individual MNLI and RTE tasks, while LoRA improves average accuracy across the two tasks for multi-tasking.RTE accuracy is relatively robust as λ changes, whereas MNLI accuracy varies substantially with λ; λ = 0 and λ = 1 recover the original RTE and MNLI LoRAs.
  • Unlearning: Negating toxic PEMs reduces GPT-2 toxicity to 0.03 for (IA)3 and 0.01 for LoRA, with LoRA representing a tenfold reduction from the 0.10 baseline.Toxic-generation ratios fall to 0.9% and 0.1%, while linguistic proficiency changes only slightly as measured by perplexity.
  • Domain transfer: LoRA significantly improves vanilla domain-transfer performance in 3 out of 4 settings, while the remaining setting is comparable to baseline.(IA)3 shows significant gains in one setting and is comparable to baselines in the other three.
  • Instruction tuning: Detoxifying Alpaca-LoRA reduces toxicity by more than 50% relatively under toxic instructions without sacrificing comparable performance on normal instructions.Manual evaluation reports win rates of 36% for toxic instructions and 27% for normal ones, with tie rates of 40% and 42%.
  • Analysis: Different PEM initializations slightly reduce improvement relative to shared initialization, but do not cause catastrophic performance drops.Merging modules trained on the same task but different distributions still outperforms the two original subset modules.

5 Discussion

The study composes trained parameter-efficient modules with linear arithmetic to manipulate capabilities across generalization, multitasking, unlearning, domain transfer, and Alpaca-LoRA detoxification. It reports successful training-free composition, while identifying inherited safety concerns and constraints on architecture, initialization, and hyperparameter selection.

  • Discussion: Linear arithmetic composes trained parameter-efficient modules for distribution generalization, multitasking, unlearning, domain transfer, and Alpaca-LoRA detoxification.The approach uses aggregation, negation, and combinations of these operations for different objectives.
  • Limitations: The work may inherit biases or safety concerns present in the existing modules, including potentially implicit toxicity not observed in the reported settings.This caveat is specifically discussed for detoxification experiments.
  • Limitations: Experiments are restricted mainly to identical module architectures and shared initialization, and the approach requires tuning the weight hyperparameter λ.Future work is proposed on varied architectures, initialization, and automatic hyperparameter selection.

A Author Contributions

The paper’s authors divide responsibilities across idea development, experimental design and execution, writing, proofreading, and project advising.

  • Methodology and Advising: Junxian He proposed the idea, refined it with Jinghan Zhang, and advised and organized the project.His advising role included initializing and organizing the overall project.
  • Experiments and Writing: Jinghan Zhang led experiments on domain transfer, LLaMA unlearning, and preliminary distribution-generalization and multitasking composition.He also co-wrote the main content with Junxian He.
  • Experiments: Shiqi Chen conducted the full unlearning experiment, while Junteng Liu conducted the main distribution-generalization and multitasking experiments.Liu’s work included extensive hyperparameter tuning.
  • Paper Writing: Other authors helped proofread the paper.

B Experimental Setup

The experiments cover distribution generalization, multitasking, unlearning, and domain transfer using multiple datasets and evaluation procedures. The setup also reports hardware, hyperparameter tables, data partitions, toxicity evaluation, and linguistic-proficiency evaluation.

  • Experimental details: The appendix reports hyperparameters for five experiments and data-distribution details for each subset.The experiments were conducted on four 3090 GPUs except the negation experiment, which used four A100 GPUs.
  • Composition for distribution generalization: Distribution-generalization experiments use eight GLUE datasets, including MNLI, RTE, CoLA, SST2, MRPC, QNLI, QQP, and STS-B.The datasets are partitioned by selecting one label and assigning most samples from that label to one subset.
  • Composition for multitasking: Multitasking experiments use prompted generative formulations for RTE and MNLI, constraining outputs to task-specific answer options.RTE uses yes/no answers, while MNLI adds maybe as an option.
  • Composition for unlearning: The unlearning setup trains on highly toxic Civil Comments data and evaluates both toxicity and linguistic proficiency.Only comments with toxicity scores exceeding 0.8 are used, and proficiency is measured with perplexity on WikiText-103.
  • Composition for domain transfer: Domain-transfer experiments use Amazon and Yelp sentiment datasets.

C Analysis on λ

The analysis examines how performance changes with the weight hyperparameter λ, varying it across a broad range and using different step sizes for selected settings.

  • Analysis on λ: Performance varies with λ, which is searched from 0 to 1 in increments of 0.02 for most experiments.The unlearning and LLaMA-extension settings use a different step size.

D Generated Examples and Ablation Results of Unlearning

The experiments compare the proposed LoRA negation approach with a simpler parameter-negation ablation in detoxification-related outputs and language-modeling behavior.

  • Table 10 compares outputs from GPT-2, toxic-Civil-Comments LoRA, and the negated-LoRA model.
  • The ablation negating all PEM parameters performs worse than the proposed negation operator for both LoRA and (IA)3.The comparison is reported in Table 11.
  • Table 11 reports output toxicity and language-modeling perplexity for the ablation analysis.

E LLaMA Experiments Details

The LLaMA experiments evaluate detoxification through generated instructions, toxicity and helpfulness metrics, human comparisons, and qualitative examples, while also illustrating arithmetic-composition behavior across tasks and domains.

  • Setup: The toxic instruction-tuning dataset contains 26792 samples generated from toxic Civil Comments using ChatGPT-assisted self-instruction.The experiment begins from the Alpaca-LoRA checkpoint trained on original Alpaca data.
  • Evaluation: The detoxification evaluation measures toxicity with Detoxify and helpfulness with GPT-4 on toxic and non-toxic instruction subsets.The test set contains 100 toxic and 100 non-toxic instructions.
  • Human evaluation: Human evaluation compares original Alpaca-LoRA and the detoxified merge module using pairwise judgments over 200 response pairs.Three evaluators first assessed 50 pairs and reached 78% tie-discounted agreement before the remaining responses were annotated.
  • Arithmetic composition experiments: Figures 5–7 vary λ to examine performance for distribution changes, task merging, and domain transfer across listed benchmarks and model configurations.Figure 6 specifically tracks MNLI and RTE validation accuracy for merged FFT and (IA)3.
  • Generated examples: The detoxified Alpaca-LoRA produces refusal-to-insult behavior despite no similar content appearing in the instruction-tuning data.
  • Experimental materials: Figures 8–10 depict the LoRA negation workflow and prompts used to generate toxic instruction-tuning data, while Figure 11 covers GPT-4 helpfulness scoring.
Loading 2306.14870v2…