PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataless Knowledge Fusion by Merging Weights of Language Models

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, Pengxiang Cheng

arXiv:2212.09849v6cs.CLcs.LG

TL;DR

The paper asks how to fuse knowledge from fine-tuned language models when their training data is unavailable. It proposes RegMean, which merges model parameters using prediction-matching weights and released statistics. Across multiple settings, RegMean outperforms merging and ensembling baselines and can improve out-of-domain generalization, while merging remains efficient.

  • Problem

    Private fine-tuning data prevents directly using multi-task learning to fuse available models, while searching across many dataset combinations can be prohibitive.

  • Method

    RegMean merges same-architecture models in parameter space using statistics such as layer-input inner-product matrices to approximate prediction matching without training data.

  • Results

    Across multiple model types and data collections, RegMean consistently and significantly outperforms model-merging and ensembling baselines and improves out-of-domain generalization over the best individual models.

  • Takeaways & Limitations

    Model merging can fuse knowledge without training data and offers an efficient alternative to hosting multiple models or training a multi-task model.

  • Takeaways & Limitations

    RegMean requires inner-product matrices; obtaining them may require a few training examples, and the statistics may reveal information about training data.

Abstract

from arXiv · show

Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns. This creates a barrier to fusing knowledge across individual models to yield a better single model. In this paper, we study the problem of merging individual models built on different training data sets to obtain a single model that performs well both across all data set domains and can generalize on out-of-domain data. We propose a dataless knowledge fusion method that merges models in their parameter space, guided by weights that minimize prediction differences between the merged model and the individual models. Over a battery of evaluation settings, we show that the proposed method significantly outperforms baselines such as Fisher-weighted averaging or model ensembling. Further, we find that our method is a promising alternative to multi-task learning that can preserve or sometimes improve over the individual models without access to the training data. Finally, model merging is more efficient than training a multi-task model, thus making it applicable to a wider set of scenarios.

1 INTRODUCTION

The paper addresses knowledge fusion when fine-tuned models are available but their private training data is inaccessible. It introduces dataless model merging as an efficient alternative for combining models across domains and tasks.

  • Motivation: Private labeled data and costly data-combination searches limit multi-task learning for building a single cross-domain model.These constraints arise from data privacy, annotation privacy, intellectual property, and the many possible source-data combinations.
  • Model Merging: Model merging combines multiple models in parameter space without access to their training data.Closed-form weight-merging algorithms avoid retraining and can therefore scale to many available datasets or model combinations.
  • Objective: The paper targets a single model that performs across multiple domains or tasks and generalizes to new domains.The evaluation protocol tests both in-domain and out-of-domain generalization.
  • Contributions: Across RoBERTa, T5, and DeBERTa, RegMean consistently outperforms model-merging and ensembling baselines and improves out-of-domain generalization over the best individual models.The reported evaluation spans multiple model types and data collections.

2 DATALESS MODEL MERGING FOR KNOWLEDGE FUSION

The dataless merging framework separates private-data model training from developer-side fusion. Released model weights and optional statistics are combined into a single model without exposing the original datasets.

  • Problem Formulation: Agents fine-tune language models from shared pretrained weights on private labeled datasets and release the resulting model weights.They may optionally disseminate statistics that do not leak information about their labeled data.
  • Problem Formulation: Developers use fine-tuned weights and released statistics as inputs to a merging function that produces a merged model.The function operates on a selected subset of models and is expected to be computationally efficient with a closed-form formulation.
  • Compared Methods: RegMean, Fisher, and Simple differ in their parameter-space merging inputs, while neither RegMean nor Fisher requires training data.RegMean uses inner-product matrices of layer inputs, whereas Fisher uses Fisher Information matrices.

3 REGRESSION MEAN FOR MODEL MERGING

RegMean derives model merging from prediction matching in linear models and extends the resulting closed-form solution to transformer linear layers. It uses activation inner-product statistics and simple averaging for remaining parameters.

  • Linear Models: RegMean minimizes the ℓ2 distance between merged-model predictions and individual-model predictions on their respective training inputs.For two linear models, the inputs are concatenated and the targets are the individual model predictions on those inputs.
  • Linear Models: The linear-model solution computes merged weights from the sum of input inner-product matrices and their weight-weighted counterparts.The closed-form solution extends to K models with minor modifications.
  • RegMean: The method is called Regression Mean and requires precomputed training-input inner-product matrices rather than recomputing the original optimization.The merger retrieves individual weights and inner-product matrices to calculate the merged weights.
  • Transformer Extension: For transformers, RegMean is applied independently to every linear layer, while embeddings and biases are merged by simple averaging.Linear-layer input features provide the activation statistics used by the method.
  • Regularization: Scaling non-diagonal inner-product entries regularizes RegMean and avoids degenerate performance in some pretrained architectures.The scalar α is usually set to 0.9; α = 0 reduces the method toward simple averaging, while α = 1 can cause performance collapse in some settings.
  • Efficiency and Privacy: RegMean computes inner-product matrices in one forward pass, making it more efficient than Fisher Information computation, which requires an additional backward pass.Its memory overhead is comparable to the model parameter count and Fisher Information matrices.
  • Efficiency and Privacy: RegMean does not require training data during merging, but released statistics can still pose a potential information-leakage risk.Agents can share low-dimensional inner-product matrices instead of private examples and labels.

4 EXPERIMENTAL SETUP

The experiments evaluate whether merging preserves performance across source domains or tasks and improves out-of-domain generalization. They compare merging, ensembling, individual-model, and multi-task-learning baselines across several model and dataset settings.

  • 4.1 EVALUATION SETTINGS: The evaluation seeks competitive performance across all source datasets and higher performance on out-of-domain test distributions.Source datasets may represent non-i.i.d. partitions, domains of one task, or different tasks.
  • 4.2 DATASETS: Experiments use GLUE, emotion-classification, and named-entity-recognition datasets to study partition, domain, and task fusion.These datasets support evaluations across multiple domains and task configurations.
  • 4.3 EXPERIMENT DETAILS: Performance is macro-averaged across incorporated domains or tasks and across out-of-domain test sets.Non-i.i.d. partition merging is evaluated on a joint-distribution test set.
  • 4.3 EXPERIMENT DETAILS: The baselines include simple averaging, Fisher-weighted averaging, logit ensembling, individual-model references, and multi-task learning.Multi-task learning trains on the joint original datasets, while ensembling averages individual-model logits before argmax.
  • 4.3 EXPERIMENT DETAILS: The study covers encoder-only RoBERTa and DeBERTa models, encoder-decoder T5, and both shared and different classification-head initialization settings.All merged models use the same architecture and pretrained initialization before fine-tuning.
  • 4.3 EXPERIMENT DETAILS: Figure 3 summarizes relative performance changes for pairwise merges across 10 or 15 domain-model combinations in emotion classification and NER.Positive values indicate improvement over domain-specific models, and the plotted y-axes use different scales.

5 RESULTS

Across in-domain, cross-task, and out-of-domain settings, RegMean generally provides the strongest dataless model-merging performance. Its benefits are substantial in pairwise domain merging, while poor individual models, pretrained-model choice, and required statistics constrain performance or applicability.

  • Experimental setup: The experiments compare dataless merging methods with individual models, multi-task learning, and ensembling across in-domain and out-of-domain settings.They cover non-i.i.d. partitions, different tasks, and different domains.
  • Merging non-i.i.d. partitions: Across eight GLUE tasks with non-i.i.d. partitions, merging consistently improves average individual-model performance, while RegMean outperforms simple averaging and matches Fisher-weighted averaging.This setup evaluates merged models on validation data combining both partitions.
  • Pairwise domain merging: 12% versus 55% and 15% versus 85% are the Emotion performance-drop rates for RegMean versus simple averaging on RoBERTa-base and DeBERTa-large, respectively.The comparison concerns pairwise merging of models trained on different domains.
  • Merging all domain-specific models: When all domain-specific models are merged, RegMean achieves the best in-domain performance in four of five evaluated model-task scenarios, with Fisher slightly better for DeBERTa-large on Emotion.RegMean also outperforms ensembling in four of five scenarios; simple averaging performs poorly except for T5.
  • Out-of-domain generalization: Knowledge fusion can hurt out-of-domain performance when poor individual models are included: no merging algorithm or MTL beats the best individual model on NER CoNLL and Twitter.Incrementally merging a subset selected using known OOD validation performance is proposed to mitigate this effect.
  • Discussion and limitations: RegMean’s behavior depends on the pretrained model and scaling parameter α: performance is stable for α = 0.1–0.9 but drops at α = 1.0.T5-base is generally easier to merge, whereas DeBERTa-large is harder to merge; inner-product statistics also impose practical and privacy constraints.

6 RELATED WORK

Prior work uses weight averaging and model merging for varied goals, while knowledge distillation typically requires access to data. These approaches span performance improvement, domain generalization, task transfer, few-shot capability, and robustness.

  • Weight averaging has been applied across checkpoints, runs, configurations, adapters, and domain experts to improve performance, generalization, few-shot capabilities, or robustness.
  • Fisher-weighted averaging merges models to improve a single target task by leveraging donor-task models.
  • Knowledge distillation fuses multiple teacher models into a smaller student but generally requires full access to data for distillation.

7 CONCLUSIONS AND FUTURE WORK

The paper presents RegMean for dataless fusion of fine-tuned language models and reports gains over alternative merging and ensembling methods in several settings. It also emphasizes efficiency and identifies broader applications and future extensions.

  • RegMean fuses multiple fine-tuned language models without training data and outperforms other dataless merging or ensembling alternatives across experimental setups.
  • Model merging can outperform individually trained models in non-i.i.d. and out-of-domain experiments.
  • Merged models are efficient, add few parameters, and have inference speed similar to an individual model, making them practical compared with hosting multiple models.
  • Future work includes merging models with different initializations or architectures, sequentially merging models at scale, and merging pre-trained models before fine-tuning.
  • Model merging may support intermediary-task selection and federated learning with models trained on private data.

A DERIVATION OF THE COMPLETE FORMULATION OF REGMEAN

RegMean formulates merging K linear models as a convex optimization problem that balances prediction fitting with closeness to individual weights. Its regularization is scaled to account for differing input correlations across models, layers, and datasets.

  • RegMean formulates merging K linear models through an optimization problem over the merged weight W.
  • The regularization term encourages merged weights to remain close to each individual model, with non-negative row-specific strengths λij.
  • The objective is convex in W, so the merged weights can be obtained by setting the gradient with respect to W to zero.
  • Regularization strengths are scaled because X_i^T X_i can differ substantially across models, layers, or datasets.
  • Relative scaling of non-diagonal inner-product terms has the same regularizing effect as increasing diagonal terms, with α = 1/(1+γ).

B DETAILS FOR DATASETS, PREPROCESSING, METRICS, AND TRAINING

The experiments evaluate merging across GLUE, emotion-classification, and NER settings using task-specific metrics and domain partitions. Implementation details include architecture-specific heads or templates and architecture-agnostic extraction of linear-layer inputs.

  • Datasets and metrics: GLUE evaluation uses Matthews Correlation for CoLA, Pearson Correlation for STS-B, and accuracy for all other tasks.
  • Datasets and preprocessing: Non-i.i.d. GLUE partitions concentrate 80% of a sampled key class in one partition while balancing partition sizes with other examples.
  • Datasets and metrics: Emotion models train on five datasets and evaluate on five held-out datasets.
  • Training and preprocessing: RoBERTa and DeBERTa use binary classification heads per class, excluding heads for classes absent from a model's training dataset.
  • Training and preprocessing: T5 reformulates classification as sequence-to-sequence prediction using yes/no outputs for each dataset class.
  • Datasets and metrics: NER models train on six OntoNotes domains and test out-of-domain on CoNLL and Twitter NER.
  • Implementation: The implementation uses PyTorch forward hooks to collect linear-layer inputs, keeping RegMean independent of model architecture.

C SENSITIVITY ANALYSIS

Sensitivity analyses examine batch counts, regularization choices, layer visualizations, and model-pair selection. Performance saturates with relatively few batches, while relative scaling and model choice materially affect results.

  • Number of batches: Performance generally improves as the number of batches increases but soon saturates around N = 100.
  • Regularization: Relative scaling of non-diagonal inner-product terms performs clearly better than adding a constant to diagonals for T5 emotion-model merging.
  • Visual analyses: Figure 7 visualizes weight-vector ℓ2 distances in intermediate and output layers of transformer-block MLPs.
  • Model selection: OOD performance depends on which RoBERTa-base emotion models are merged; combining TEC and ISEAR gives the best OOD performance in the reported pairs.

D PERMUTATION MATCHING ALGORITHMS FOR MERGING LANGUAGE MODELS

The permutation-matching analysis compares weight-based and activation-based matching for models sharing pretrained initialization. Weight distances indicate identity matching, whereas activation-based permutations reduce performance below simple averaging.

  • Weight-Based Matching: Weight-based matching computes pairwise ℓ2 distances between corresponding weight vectors and uses them as optimal-transport ground metrics.
  • Weight-Based Matching: In the examined MLP layers, diagonal weight distances are much smaller than non-diagonal distances, yielding a trivial identity permutation.
  • Weight-Based Matching: The authors conjecture that shared pretrained initialization stabilizes training and prevents weight permutations, with residual connections possibly reinforcing this effect.
  • Activation-Based Matching: Activation-based matching constructs a similarity matrix from activation inner products and solves a linear assignment problem for permutations.
  • Activation-Based Matching: Activation similarity matrices are far from diagonal, but applying their permutations produces performance far below simple averaging.
  • Activation-Based Matching: The paper conjectures that activation permutations may not faithfully represent weight permutations in this shared-initialization setup.
Loading 2212.09849v6…