PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossMean: Model Merging by Pairing Merged and Expert Activations

Moshe Eliasof

arXiv:2610.14762v1cs.LG

TL;DR

Closed-form merging uses activation statistics measured on single experts, although merged layers receive inputs from merged prefixes, and recent re-measurement methods still use statistics from one network. CrossMean pairs merged and expert activations in a bilinear statistic and derives a label-free alignment criterion; it improves over expert-statistic merging across tested vision and BERT settings and over depth-ordered correction on tested backbones when all linear layers are merged.

  • Problem

    Closed-form methods measure each layer’s activation statistics on a single expert, although that layer receives inputs from the merged prefix, and re-measurement methods still form statistics from one network.

    Source: §1 · p. 1

  • Method

    CrossMean relaxes the equal-matrix linear-system formulation by pairing merged and expert inputs in a bilinear statistic, and derives a label-free alignment criterion from the same statistics.

    Source: §1 · pp. 1–2

  • Results

    CrossMean improves on RegMean across eight-task sets of five vision backbones and BERT encoders, and with all linear layers merged also improves on depth-ordered correction on tested backbones; lower criterion values correspond to larger improvements.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results suggest that merging benefits from pairing merged and expert activations.

    Source: §6 · p. 8

Abstract

from arXiv · show

Model merging combines fine-tuned experts into one network without retraining, and closed-form methods compute each merged layer from statistics of its input activations. However, these statistics are measured on a single expert, while in the merged model, each layer receives activations from the merged layers that precede it. Recent work re-measures these statistics in the merged model but still forms them from a single network. We formulate closed-form methods as a linear system and show that they use the same matrix on both sides. We then introduce CrossMean, which replaces this matrix on one side with a bilinear statistic that pairs merged and expert activations. From the same statistics, we also derive a label-free criterion that indicates the task sets in which accounting for the merged activations improves the merge more. Empirically, task sets with a lower criterion value show a larger improvement. On the eight-task benchmark of each vision backbone that we tested and on BERT encoders, CrossMean improves on merging with statistics measured on the experts, and with all linear layers merged, it also improves on re-measuring these statistics in the merged model. These results suggest that merging benefits from pairing merged and expert activations.

1 Introduction

Closed-form merging measures activations on individual experts even though merged layers receive inputs from preceding merged layers. CrossMean pairs merged and expert inputs, and a label-free criterion tracks where accounting for merged inputs improves merging more.

1 Introduction

  • Existing methods measure statistics on a single network, although merged-layer inputs come from the preceding merged layers, creating a mismatch that can propagate to subsequent layers.The merged layer should map merged inputs to outputs produced by each expert on its expert inputs; a Gram matrix represents only one input distribution.Source: §1 · p. 1
  • A linear-system formulation shows that existing methods use the same matrix on both sides, with activation-based methods using a single-network Gram matrix.The linear system obtains merged weights by weighting expert weights with one matrix and normalizing by the sum of a second matrix.Source: §1 · p. 1
  • CrossMean replaces the Gram matrix on one side with a bilinear statistic pairing merged and expert inputs.The cross statistic is averaged along a path from the merged prefix to the expert; CrossMean contains RegMean and Progressive RegMean as special cases.Source: §1 · p. 2, Figure 1 · p. 2
  • The label-free alignment criterion is lower where Progressive RegMean and CrossMean improve more on RegMean.It is computed from statistics involving merged inputs and can be calculated before the closed-form merge.Source: §1 · p. 2
  • CrossMean improves on RegMean across eight-task sets for five vision backbones and BERT encoders, and with all linear layers merged it also improves on the depth-ordered correction on tested backbones.Task sets with lower criterion values show larger improvements.Source: §1 · p. 2

2 Related Work

Related work spans data-free weight-space methods, statistics-weighted closed-form merges, gradient-based or test-time approaches, and depth-ordered corrections. CrossMean differs from these methods by pairing merged and expert inputs in a bilinear statistic.

2 Related Work

  • Weight averaging and Task Arithmetic combine experts in weight space; TIES, DARE, Model Stock, and Iso-C reduce interference or rescale the sum.These methods do not use activation statistics and form data-free baselines.Source: §2 · p. 3
  • Fisher merging and RegMean use statistics measured on a single expert, while CrossMean pairs merged and expert inputs.MaTS casts Fisher merging and RegMean as a common linear system; LOT uses the same matrices as RegMean.Source: §2 · p. 3
  • AdaMerging, E-WEMoE, Twin-Merging, and ProDistill use test-time tuning, routing, or distillation rather than the paper’s closed-form approach.These approaches rely on gradient steps; the paper studies statistics computed on a small unlabeled calibration set.Source: §2 · p. 3
  • RegMean++ and Chain of Merges (Buzzega et al., 2025) re-measure inputs on merged prefixes in depth order, but keep the linear-system matrices equal.CrossMean instead replaces the right-hand-side Gram matrix with a bilinear statistic and uses its criterion to anticipate improvements over RegMean.Source: §2 · p. 3

3 Closed-Form Merging and CrossMean

The section formulates closed-form merging as a per-layer linear system, identifies the merged-prefix mismatch, and introduces CrossMean to pair merged and expert inputs. It also specifies the path averaging used by CrossMean and the layer coverage and implementation choices considered.

3.1 Setup and notation

  • The setup distinguishes MLP coverage, which averages attention projections, from full coverage, which also merges attention input and output projections in closed form.Full coverage includes every linear layer of the transformer.Source: §3.1 · p. 3

3.2 Closed-form merging methods as a linear system

  • Existing methods use the same matrix on both sides of the per-layer linear system, up to a scalar for weight-space methods.For activation-based methods, the matrix is a Gram matrix measured on a single network.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 The merged-prefix mismatch

  • Progressive RegMean measures a Gram matrix on merged-prefix inputs but uses that same matrix on both sides, leaving the relation to expert inputs unrepresented.It corrects where the Gram matrix is measured, rather than pairing merged and expert inputs.Source: §3.3 · p. 4

3.4 CrossMean: Bilinear Statistic Merging

  • CrossMean replaces the right-side Gram matrix with a bilinear statistic pairing merged and expert inputs.It keeps the merged-input Gram matrix on the left side of the system.Source: §3.4 · p. 4
  • CrossMean keeps merged-prefix inputs as the layer input while targeting the expert’s output on expert inputs.This avoids Progressive RegMean’s restriction that the merged and expert inputs must coincide for its target to match the expert output.Source: §3.4 · pp. 4–5
  • The cross statistic can be asymmetric and non-positive-semidefinite; the system only inverts the regularized merged-input Gram matrix.Removing the bilinear statistic’s antisymmetric part lowers accuracy on both tested backbones.Source: §3.4 · p. 5
  • CrossMean averages cross statistics along an interpolation path from the merged prefix to the expert, using S = {0, 1/2, 1}.The sampled points include both endpoints and the midpoint.Source: §3.4 · p. 5
  • Progressive RegMean and CrossMean reduce statistic off-diagonals and add a ridge term scaled by the mean diagonal of the matrix being inverted.Source: §3.4 · p. 5

4 The Alignment Criterion

The alignment criterion compares the merged-input Gram matrix with the cross statistic to characterize merged-prefix mismatch. It can be computed before the closed-form merge, and lower values are associated with larger improvements for the depth-ordered correction on the reported vision task sets.

From the linear system to two statistics

  • The criterion is built from the merged-input Gram matrix and the cross statistic, which capture the statistics used by Progressive RegMean and CrossMean.When merged and expert inputs are equal, the statistics coincide and the three linear systems match.Source: §4 · p. 5

The alignment criterion

  • The criterion ρ averages the Frobenius cosine between the merged-input Gram matrix and the cross statistic across mergeable layers and experts.It measures their alignment independently of scale; ρ = 1 when the merged inputs equal the expert inputs.Source: §4 · p. 6

Computing the criterion before the closed-form merge

  • ρ can be computed before the closed-form merge using experts and calibration sets, without labels or test data, at the cost of one additional forward pass per calibration set.Source: §4 · p. 6

Comparison with the raw prefix drift

  • On eight-task sets for ViT-B/32, ViT-B/16, and ViT-L/14, lower ρ is associated with larger improvement from the depth-ordered correction, while raw prefix drift does not order improvements.The authors expect improvements from CrossMean and the depth-ordered correction over RegMean to decrease as ρ approaches 1.Source: §4 · p. 6

5 Experiments

The experiments evaluate CrossMean against activation-based and other merging baselines on vision task sets, test whether the label-free criterion ρ tracks improvement, and assess transfer to BERT encoders. CrossMean improves on RegMean across the reported vision and BERT settings, while lower-ρ task sets show larger gains.

5 Experiments

  • The experiments use five CLIP vision backbones, task sets of eight, fourteen, or twenty tasks, and 256 unlabeled calibration images per task; test inputs are excluded from merging and selection.The shared-protocol grid contains fifteen backbone–task-set settings, and reported main-text results average three calibration seeds.Source: §5 · p. 6

5.1 Merging accuracy of CrossMean

  • With full coverage, CrossMean and LOT + CrossMean improve on all activation-based baselines, including RegMean++ and LOT, across three backbones.CrossMean also exceeds Progressive RegMean by +0.94 to +1.90% on all three tested backbones.Source: Table 3 · p. 7, §5.1 · p. 7
  • CrossMean improves on RegMean on all five vision backbones and has higher accuracy than every compared method in eleven of fifteen shared-protocol settings.With MLP coverage, the comparison covers the eight-task sets; the shared-protocol grid spans five backbones and three task-set sizes.Source: §5.1 · p. 7

5.2 The alignment criterion and the improvement of CrossMean

  • In controlled pairs across five backbones, lower-ρ task sets yield CrossMean-over-RegMean margins of +7.01% to +12.50%, versus at most +1.70% for higher-ρ sets.Across fifteen shared-protocol settings and ten controlled-pair task sets, lower ρ is associated with a larger CrossMean-over-RegMean margin (rank correlation −0.72).Source: §5.2 · pp. 7–8

5.3 Merging BERT encoders

  • On BERT-base encoders, CrossMean improves on RegMean by +5.0 to +7.2% in all four GLUE settings, and GLUE-6 has the larger margin at N = 256.Progressive RegMean scores below RegMean in these settings.Source: §5.3 · p. 8

6 Conclusion

The paper analyzes the statistics used by closed-form merging and introduces CrossMean to pair merged and expert inputs. Experiments find gains over RegMean on vision and BERT, with larger improvements on task sets having lower criterion values.

6 Conclusion

  • CrossMean replaces the shared Gram matrix in the linear system with a bilinear statistic pairing merged and expert inputs.The paper derives a label-free alignment criterion from the same statistics.Source: §6 · p. 8
  • CrossMean improves on RegMean across the eight-task sets of five vision backbones and on BERT encoders.With all linear layers merged, it also improves on the depth-ordered correction on each tested backbone.Source: §6 · p. 8
  • Task sets with lower values of the alignment criterion show larger improvements, and the results suggest merging benefits from pairing merged and expert activations.Source: §6 · p. 8

A Protocol, experts, baselines, and further related work

The appendix specifies the shared evaluation protocol, model and merge settings, and implementation details. It also distinguishes this study’s merged-prefix mismatch from related work on merging, activation correction, and representation comparison.

A Protocol, experts, baselines, and further related work

  • The shared-protocol grid evaluates fifteen backbone–task-set combinations using 256 unlabeled calibration images per task, validation for hyperparameter selection, and disjoint test splits.Every method uses the same protocol; RegMean, Progressive RegMean, and CrossMean have no scale hyperparameter.Source: A Protocol, experts, baselines, and… · p. 13
  • Vision experts span five CLIP backbones and nested task sets of eight, fourteen, and twenty tasks; the BERT experiments merge fine-tuned BERT-base encoders on GLUE.Public and newly fine-tuned experts are used across the vision settings.Source: A Protocol, experts, baselines, and… · p. 13
  • MLP coverage merges the two MLP linear layers per block, whereas full coverage also merges attention input and output projections; normalization parameters, biases, and embeddings are averaged.Query, key, and value projections share an input and are solved with a shared Gram matrix.Source: A Protocol, experts, baselines, and… · p. 13
  • Progressive RegMean re-measures layer inputs on the merged prefix in depth order, while CrossMean uses a bilinear statistic pairing merged and expert inputs.Progressive RegMean requires more forward passes than RegMean; CrossMean’s total is at most 3LT for L mergeable layers and T tasks.Source: A Protocol, experts, baselines, and… · p. 14

A.1 Further related work

  • Unlike methods that match units in independently trained networks, this study concerns experts sharing a pretrained model and the resulting merged-prefix mismatch.Its label-free criterion compares the merged-input Gram matrix with the merged–expert cross statistic, unlike transfer estimators that use target-task labels.Source: §A.1 · p. 15

B Additional merging results

CrossMean leads the comparison in most shared-protocol settings, improves on baselines for several individual tasks, and gains further under full coverage.

B Additional merging results

  • On each of five backbones, CrossMean improves on all baselines for RESISC45, EuroSAT, SVHN, GTSRB, and DTD.The per-task results are reported in Tables 9 and 10.Source: B Additional merging results · p. 16
  • CrossMean reaches higher accuracy than every other compared method in 11 of 15 shared-protocol settings.Table 7 reports test accuracies with GELU and MLP coverage, averaged over three seeds.Source: Table 7 · p. 16, B Additional merging results · p. 16
  • With full coverage, CrossMean improves on both RegMean++ forms on ViT-B/32, ViT-B/16, and ViT-L/14.The two RegMean++ forms reach accuracies within 0.1% of each other on all three backbones.Source: B Additional merging results · p. 16

B.1 Applying CrossMean to LOT Merging

CrossMean is applied to LOT by changing its statistics while retaining its closed-form solution. The resulting LOT + CrossMean improves on LOT across all three tested backbones.

B.1 Applying CrossMean to LOT Merging

  • LOT + CrossMean and LOT + Progressive RegMean retain LOT’s closed-form solution and normalization scales, changing only the statistics used to compute it.LOT + CrossMean uses the CrossMean objective along points on the path from the merged prefix to each expert.Source: §B.1 · p. 17
  • With full coverage, CrossMean and Progressive RegMean both exceed LOT on all three tested backbones.LOT fits each linear layer to reproduce expert features on expert inputs; its relation to RegMean holds on calibration-data input directions.Source: §B.1 · p. 17
  • LOT + CrossMean improves on LOT by +3.75%, +2.11%, and +0.50% on B/32, B/16, and L/14, respectively.The improvement occurs in every seed on all three tested backbones.Source: §B.1 · p. 17, Table 11 · p. 19
  • On each of the three backbones, LOT + CrossMean improves more on LOT than LOT + Progressive RegMean.The improvement of LOT + CrossMean over LOT decreases as the task-set alignment criterion ρ increases.Source: §B.1 · p. 17, Table 11 · p. 19

C Additional results on the alignment criterion

Across task sets, lower alignment-criterion values are associated with larger merging margins. The appendix also compares the criterion with raw prefix drift and describes its layerwise behavior.

C Additional results on the alignment criterion

  • Across 25 task sets, lower ρ is associated with larger margins of CrossMean and Progressive RegMean over RegMean, with rank correlations of −0.72 and −0.73, respectively.Table 12 includes the fifteen shared-protocol settings and ten controlled-pair task sets, sorted by ρ.Source: Table 12 · p. 20, C Additional results on the alignment… · p. 20
  • Unlike Iso-C’s alignment, which measures task-matrix alignment with the merged matrix’s dominant singular subspace, ρ compares two activation statistics.The compared statistics are the Gram matrix of merged inputs and the cross statistic; ρ anticipates improvements over RegMean.Source: C Additional results on the alignment… · p. 20
  • On the three eight-task sets, raw prefix drift does not order Progressive RegMean’s margins, whereas ρ increases as those margins decrease.The drift values are 0.576, 0.554, and 0.587, while ρ values are 0.894, 0.919, and 0.931 for ViT-B/32, B/16, and L/14, respectively.Source: C Additional results on the alignment… · pp. 20–21
  • Raw drift weights every direction equally, including low-energy directions, while the criterion weights directions by the energy merged inputs place in them.The distinction is described through the input-direction weights used by the two measures.Source: C Additional results on the alignment… · p. 21
  • Layerwise ρ varies with depth differently across backbones, and the criterion is the average of the layerwise values.Figure 2 marks ρ = 1 as the value when merged inputs equal expert inputs.Source: C Additional results on the alignment… · p. 21, Figure 2 · p. 21

C.1 Accounting for task difficulty

The difficulty study tests whether ρ’s association with merging margins is distinct from task difficulty, then reports correlation uncertainty and introduces further design-ablation and cost analyses.

C.1 Accounting for task difficulty

  • The study draws twelve random eight-task sets on ViT-B/32 and measures margins, ρ, and task difficulty as mean expert accuracy on its own task.This design addresses the possibility that ρ reflects task difficulty rather than merged-prefix mismatch.Source: §C.1 · pp. 21–22
  • On ViT-B/32, lower ρ is associated with larger Progressive RegMean margins, including after adjusting for task difficulty.Across twelve task sets, the unconditional rank correlation is −0.73 and the partial correlation is −0.71; the partial-correlation interval excludes zero.Source: Figure 3 · p. 21, §C.1 · p. 22
  • On ViT-B/16, ρ and task difficulty retain their signs and relative strength; ρ’s unconditional rank correlation with the margin is −0.64 and is significant.After adjusting for difficulty, the corresponding ρ correlation is −0.61; the difficulty correlations are +0.38 and +0.33.Source: §C.1 · p. 22

C.2 Uncertainty of the rank correlations

  • The reported intervals support the adjusted ViT-B/32 association, the significant unconditional ViT-B/16 association, and pooled correlations over 25 task sets with p-values at most 2 × 10^-4.Table 13 reports percentile-bootstrap 95% intervals and two-sided permutation p-values, computed from 104 resamples.Source: §C.2 · p. 22, Table 13 · p. 22

D CrossMean: design ablations and cost

  • The design-ablation and cost section covers CrossMean design ablations, computational cost, and the effect of calibration-set size.The passage identifies these topics but does not report their findings.Source: D CrossMean: design ablations and cost · p. 23

D.1 Design ablations

The ablations identify the bilinear statistic’s asymmetry as important: symmetrizing it lowers CrossMean’s margin over Progressive RegMean, while the other tested changes have little effect. CrossMean also uses less peak memory than RegMean, but takes longer and reaches its calibration-image plateau later.

D.1 Design ablations

  • At β = 1, symmetrization lowers the margin over Progressive RegMean to −70.61% on ViT-B/32 and −78.82% on ViT-L/14.The margin decreases monotonically as β increases on both backbones.Source: Table 14 · p. 23, §D.1 · p. 23
  • Shrinkage toward the Gram matrix and matched-pair token dropout each change the margin by at most 0.97% without changing its sign on either backbone.By comparison, symmetrization lowers the margin by 26 to 78%.Source: §D.1 · p. 23, Table 14 · p. 23

D.2 Cost and calibration set size

  • CrossMean takes 31.2× RegMean’s time on ViT-B/32 and 68.8× on ViT-H/14, while Progressive RegMean takes 24.4× and 25.9×, respectively.The relative time of CrossMean depends on the backbone rather than task count; timings include feature-extraction forward passes.Source: Table 15 · p. 23, §D.2 · p. 24
  • CrossMean and Progressive RegMean need 28 to 34% less peak memory than RegMean on both reported backbones.They avoid holding all experts’ Gram matrices at once.Source: Table 15 · p. 23, §D.2 · p. 24
  • RegMean and Progressive RegMean plateau at 64 calibration images per task, while CrossMean plateaus at about four times that number and improves on Progressive RegMean from 32 onward.The 256 images per task used in the vision experiments lie on CrossMean’s plateau.Source: Figure 4 · p. 24, §D.2 · p. 24

E Merging BERT encoders on GLUE

The BERT experiments evaluate CrossMean and related merges on GLUE-4 and GLUE-6 using fine-tuned BERT-base encoders. The lower-criterion task set has the larger reported CrossMean improvement over RegMean at N=256, and results are averaged over three calibration draws.

Merges and evaluation

  • The experiments merge fine-tuned BERT-base encoders on GLUE-4 and GLUE-6, retaining each task’s expert classification head on the merged encoder’s pooled output.The encoder merge includes the pooler; selected merges are evaluated on the GLUE validation split, with Matthews correlation for CoLA and accuracy for other tasks.Source: E Merging BERT encoders on GLUE · p. 24

Calibration draws

  • Reported results average three fixed-seed calibration draws; standard deviations are at most 0.35 for CrossMean and 1.21 for Progressive RegMean.Margins over RegMean are paired within each calibration draw.Source: E Merging BERT encoders on GLUE · p. 24

The criterion

  • At N=256, the lower-ρ task set has the larger CrossMean-over-RegMean improvement: +7.24% versus +5.11%.ρ is 0.9417 on GLUE-4 and 0.9358 on GLUE-6, and changes by at most 0.0011 across calibration set sizes.Source: E Merging BERT encoders on GLUE · p. 24

Related papers

Loading 2610.14762v1…