PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Layer-wise Model Merging through Chain of Merges

Pietro Buzzega, Riccardo Salami, Angelo Porrello, Simone Calderara

arXiv:2508.21421v3cs.LG

TL;DR

Layer-wise model merging can overlook dependencies between layers, creating distributional mismatches for methods that rely on activations. CoM sequentially merges parameters while updating activation statistics, and it outperforms existing methods on LoRA fine-tuning while matching state of the art on traditional full-rank checkpoints.

  • Problem

    Model merging combines specialized task-specific checkpoints without retraining, but layer-wise methods can overlook inter-layer dependencies and produce distributional mismatches.

    Source: §1 · pp. 1–2

  • Method

    CoM merges parameters sequentially from the input layer onward and updates activation statistics using outputs from the partially merged model.

    Source: §1 · p. 2

  • Results

    CoM outperforms existing methods by a large margin on LoRA fine-tuning and matches state of the art on traditional full-rank checkpoints.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CoM provides a framework applicable to any activation-based merging methodology.

    Source: §1 · p. 2

    CoM requires representative data samples to estimate activation statistics, which may raise privacy, licensing, or governance concerns when task data cannot be shared or centrally processed.

    Source: Impact statement · p. 9

Abstract

from arXiv · show

Fine-tuning pretrained models has become a standard pathway to achieve state-of-the-art performance across a wide range of domains, leading to a proliferation of task-specific model variants. As the number of such specialized models increases, merging them into a unified model without retraining has become a critical challenge. Existing merging techniques operate at the level of individual layers, thereby overlooking the inter-layer dependencies inherent in deep networks. We show that this simplification leads to distributional mismatches, particularly in methods that rely on intermediate activations, as changes in early layers are not properly propagated to downstream layers during merging. We identify these mismatches as a form of internal covariate shift, comparable to the phenomenon encountered in the initial phases of neural networks training. To address this, we propose Chain of Merges (CoM), a layer-wise merging procedure that sequentially merges weights across layers while sequentially updating activation statistics. By explicitly accounting for inter-layer interactions, CoM mitigates covariate shift and produces a coherent merged model through a series of conditionally optimal updates. Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance.

1. Introduction

Model merging combines specialized checkpoints without retraining, but independently merging layers can overlook dependencies between them. The introduction identifies merging covariate shift as a problem for activation-based methods and presents Chain of Merges (CoM) as a sequential approach.

1. Introduction

  • Model merging combines task-specific checkpoints without retraining, but independently trained models are not guaranteed to combine seamlessly.Source: §1 · p. 1
  • Changing early-layer parameters can shift output activations, giving downstream layers unexpected inputs and allowing mismatches to accumulate through the network.Source: §1 · p. 1
  • Activation-statistic methods merge layers simultaneously using pre-merge activations, so they do not account for distribution shifts caused by preceding merged layers.Source: §1 · p. 2
  • CoM sequentially merges parameters from the input layer onward and updates activation statistics using outputs from the partially merged model.The introduction casts layer-wise parameter merging as distillation that replicates task-specific activation distributions.Source: §1 · p. 2
  • CoM outperforms existing methods by a large margin on LoRA fine-tuning and matches state of the art on traditional full-rank checkpoints.Source: §1 · p. 2

2. Background

The background describes merging models with identical architectures and introduces RegMean as a method for approximating their layer behavior with a shared transformation.

2. Background

  • The setup considers N models with identical architectures, trained independently on distinct datasets, and targets their L linear layers for merging.Source: §2 · p. 2
  • Transformer linear projections constitute approximately 95% of the parameter count in the architectures considered.Source: §2 · p. 2
  • RegMean seeks one linear transformation that approximates the original layers on their respective inputs, with a closed-form solution for the merged layer.Source: §2 · p. 2

3. Methodology

Independent layer merging can leave later-layer activation statistics mismatched with the inputs produced by earlier merged layers. Chain of Merges updates those statistics recursively and adds task- and layer-specific importance weighting.

3. Methodology

  • Merging covariate shift arises when merging an earlier layer changes later-layer inputs, making the statistics used to merge subsequent layers inconsistent with their actual inputs.The mismatch is analogous to internal covariate shift and is especially relevant when merging relies on activation statistics.Source: §3 · p. 2

3.1. Chain of Merges

  • CoM starts with the first layer’s raw inputs, then propagates activations from merged layers forward and updates each subsequent layer’s merged weights using those activations.At each stage, the statistics reflect the accumulated effect of preceding merges, mitigating merging covariate shift without extra cost beyond Eq. 4.Source: §3.1 · pp. 2–3

3.2. Correlation-based importance

  • The merged weight matrix is an approximation, not an exact reconstruction, because fixed-dimensional compression discards representational capacity and leaves residual regression error.That residual error perturbs activations across tasks.Source: §3.2 · p. 3
  • Task- and layer-specific importance weights bias merging toward retaining tasks considered more vulnerable to performance loss after merging.The method associates greater importance with tasks whose features are more distant from pretraining representations, while weakly correlated features suggest greater proximity.Source: §3.2 · pp. 3–4
  • CoM computes task importance from merged-model features, whose layer-input Gram matrices are already available during merging.It uses a correlation matrix derived from normalized features in place of the Gram matrix because matrix conditioning affects numerical stability.Source: §3.2 · pp. 3–4

4. Experimental Study

The evaluation covers static model merging across vision and language benchmarks, comparing CoM with established methods under LoRA and full fine-tuning. CoM leads the LoRA comparisons and remains highly effective with full-rank checkpoints, where its margin over competing methods is smaller.

4.1. Evaluation settings

  • The evaluation uses static merging across eight vision datasets and six language datasets, with a single merged backbone and no task-specific routing at inference.Vision uses CLIP zero-shot heads and language uses fine-tuned heads; the study includes ViT-B/32, ViT-L/14, and Llama 3-8B checkpoints.Source: §4.1 · pp. 4–5

4.2. Results

  • CoM surpasses CoreTSV by more than 16 percentage points on ViT-B/32 LoRA vision tasks.CoreTSV is the strongest baseline and consistently ranks second.Source: Table 1 · p. 5, §4.2 · p. 5
  • 91.06% versus 86.21% for the second-best method, CoM's ViT-L/14 LoRA result remains state of the art.The larger backbone raises nearly all methods and places median normalized accuracy around 75%.Source: Table 1 · p. 5, §4.2 · p. 5
  • On six language benchmarks, CoM delivers near-perfect merged performance and surpasses the strongest baseline by more than 5 points.Task Arithmetic already reaches 90.38% average normalized accuracy; FisherAVG and MaTS are omitted because their Fisher Information Matrix is prohibitively expensive for Llama3-8B.Source: §4.2 · p. 5
  • The results section also examines CoM's accuracy across sample counts and the individual impact of MCS, activation normalization, and feature-correlation weighting.The supplied table descriptions identify these analyses but do not provide their cell values.Source: Table 2 · p. 6, Table 4 · p. 6
  • With full fine-tuning, CoM achieves 94.8% normalized and 87.7% absolute accuracy on ViT-B/32, and remains top-performing on ViT-L/14.The smaller margin over competing methods suggests closed-form activation matching is less effective for full-rank checkpoints.Source: Table 3 · p. 6, §4.2 · p. 6

5. Model Analysis

The analysis finds that merging covariate shift grows across layers, and examines sample needs, component contributions, feature-correlation weighting, and computational cost.

  • With 5 samples CoM surpasses the current state of the art, and with 100 it approaches near-maximal performance.The study also reports numerical stability with as few as 2 samples per task.Source: §5 · p. 6
  • Addressing merging covariate shift alone is sufficient for state-of-the-art performance; off-diagonal-norm weighting helps vision more than language, while activation normalization consistently improves results.The authors attribute the difference in weighting impact to greater task and domain similarity in language than in vision.Source: §5 · pp. 6–7
  • Merging covariate shift appears across layers and tends to increase with depth as earlier layers influence subsequent ones.It is measured by modeling activation outputs as multivariate Gaussians and using Earth Mover’s distance.Source: Figure 1 · p. 7, §5 · p. 7
  • Across datasets and architectures, normalized task-performance gaps and layer-averaged correlation weights exhibit a consistent correlation.The performance gap compares each task-specific model with the pretrained model’s zero-shot performance.Source: Figure 2 · p. 7, §5 · p. 7
  • For ViT-B/32, CoM has a resource profile comparable to other state-of-the-art methods and exceeds RegMean in computational efficiency.CoM’s merged-activation update does not increase the number of forward passes; CoM_fast uses 5 samples and CoM_best uses 500 for vision and 300 for NLP.Source: §5 · p. 7

6. Comparison with Related Work

The comparison covers parameter interpolation and interference-reduction heuristics, parameter- and feature-alignment methods, and dynamic routing approaches distinct from this paper’s static setting.

7. Conclusions

The paper identifies merging covariate shift in activation-statistics methods and proposes sequentially updating activation statistics to address it. It reports consistent CoM outperformance on standard vision and language benchmarks, while noting data and objective limitations.

  • CoM updates activation statistics autoregressively to capture inter-layer dependencies and mitigate merging covariate shift.The paper reports that this shift emerges when activation-statistics methods merge layers independently.Source: §7 · p. 8
  • CoM consistently outperforms existing methods across standard vision and language benchmarks, architectures, and domains.Source: §7 · p. 8
  • The approach relies on validation samples and focuses on activation matching; the authors identify generative solutions and other merging objectives as future directions.Source: §7 · p. 8

Impact statement

The impact statement describes potential deployment and reuse benefits alongside safety, privacy, and governance risks. The appendix derivation solves for the merging weights and specifies a pseudoinverse when the relevant matrix is singular.

Impact statement

  • If broadly adopted, effective merging could reduce the need to store, serve, and repeatedly fine-tune separate task checkpoints, lowering deployment and maintenance costs.The statement also describes potential energy savings and more modular reuse in resource-constrained settings.Source: Impact statement · p. 9
  • Merging may propagate undesirable behaviors from constituent models, and estimating activation statistics requires representative data that may raise privacy, licensing, or governance concerns.The authors recommend provenance and dataset governance, safety and bias evaluation of merged models, and privacy-preserving variants when sharing is constrained.Source: Impact statement · p. 9

A. Derivation of the correlation-weighted merging solution

  • The derivation sets the objective’s W_l derivative to zero and solves for W_l; if the matrix is singular, it calls for the Moore–Penrose pseudoinverse.Source: A. Derivation of the… · p. 13

B. Supplementary Ablations for Textual Datasets

On textual tasks, importance and correlation vary little, weakening their correlation and limiting the benefit of correlation-weighted merging. The measured merging covariate shift is broadly comparable to that in vision models.

B. Supplementary Ablations for Textual Datasets

FLOPs Cost (ViT-L/14)

C. Computational and Memory Cost

CoM’s computational and memory costs are evaluated against established methods, while the task-importance analysis motivates correlation weighting and tests the merged model as a reference. The reported coefficient analysis finds close alignment between CoM and zero-shot estimates.

C. Computational and Memory Cost

  • The cost evaluation reports FLOPs and theoretical memory overhead; CoM_fast uses five samples for state-of-the-art performance, while CoM_best uses the optimal sample count for maximum performance.The supplied excerpt does not state the comparative cost values.Source: C. Computational and Memory Cost · p. 14

D. Analysis of Task Importance Coefficients

  • CoM coefficients align closely with zero-shot coefficients for ViT-B and ViT-L, and both correlation-based measures track the performance-based oracle’s general trends, including high-importance EuroSAT and SVHN.Source: Figure 7 · p. 15

C. Computational and Memory Cost

  • CoM improves accuracy over RegMean while keeping forward-pass counts unchanged and using fewer examples, yielding lower computational cost.RegMean computes Gram matrices from task-specific models; CoM makes the same number of passes on the merged model.Source: C. Computational and Memory Cost · p. 15
  • CoM is reported to balance efficiency and performance, maintaining state-of-the-art accuracy with limited computation and memory overhead.Source: C. Computational and Memory Cost · p. 15

D.1. Theoretical Motivation: Orthogonality and Generalization

  • The analysis treats high off-diagonal correlations in the input Gram matrix as evidence that a task concentrates on a narrow subspace and diverges from pretraining.The method uses deviations from the approximately decorrelated pretrained representation as a signal of task specificity.Source: §D.1 · p. 15

D.2. Ablation on Reference Models

The reference-model ablation compares performance-based importance with correlations computed from the pretrained and merged models. CoM’s merged-model coefficients align closely with zero-shot coefficients, while the supplied accuracy tables cover three model settings.

E. Accuracies on each dataset

D.2. Ablation on Reference Models

  • A task-agnostic reference is needed for comparable correlation coefficients across tasks and layers; CoM uses the merged model as an efficient proxy for the zero-shot pretrained model.The authors characterize the merged model as approximately task-agnostic and balanced across tasks.Source: §D.2 · pp. 15–16
  • CoM coefficients align closely with zero-shot coefficients, and both correlation measures track the oracle’s general trends, including high-importance EuroSAT and SVHN.The comparison covers ViT-B and ViT-L; the authors report no significant deviation from the ideal pretrained reference.Source: §D.2 · p. 16

Related papers

Loading 2508.21421v3…