Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

1 Introduction

• Existing heterogeneous approaches often add distillation, adapters, learned latent spaces, routing, or alignment, leaving direct averaging across substantially different billion-parameter models unresolved.Homogeneous merging more commonly assumes shared architecture, parameter shapes, and at least partially aligned representations.Source: §1 · p. 2
• The method expands the smaller checkpoint into the larger parameter space or truncates the larger checkpoint into the smaller space, then directly interpolates without training or semantic alignment.It also omits distillation data, adapters, routing, and learned latent encoders.Source: §1 · p. 2
• Across Qwen-family model pairs, small-ratio interpolation can improve over strong source checkpoints, while near-balanced mixtures often collapse and task-level gains can coexist with regressions.For Qwen2.5-14B and Qwen2.5-32B, average scores are 0.7044 and 0.7424 for the source checkpoints, while the best merged model reaches 0.7525.Source: §1 · pp. 2–3
• The paper frames controlled weighted fusion as a strong baseline whose failure regimes may expose compatibility bottlenecks that more complex methods cannot easily bypass.Its contributions include ratio-sensitive diagnostics of preservation, transfer, collapse, and task-level seesaw behavior.Source: §1 · pp. 2–3

2 Related Work

• Homogeneous merging methods include weight averaging, task arithmetic, Fisher-weighted averaging, RegMean, sparse-mask approaches, and MoE-style routing, but typically assume compatible parameter layouts.The cited work also describes homogeneous merging as relying on similar representations and often careful tuning.Source: §2 · p. 3
• Subspace methods localize or match task-relevant parameter regions, whereas this paper deterministically projects a larger checkpoint into a smaller space before small-ratio interpolation.The related work contrasts this truncate-then-merge protocol with methods that estimate supports or matching subspaces for shape-compatible task vectors.Source: §2 · p. 3
• Heterogeneous methods use distillation, trained adapters, generated data, modular skill storage, or learned latent spaces, adding optimization, data, or trained modules to checkpoint reuse.Examples include knowledge distillation (Hinton et al., 2015), AdapterFusion (Pfeiffer et al., 2021), FuseLLM and FuseChat (Wan et al., 2024a,b), GraftLLM (Du et al., 2025a), and LS-Merge.Source: §2 · p. 3
• Training-free alignment and repair methods address mismatch through permutation matching, activation repair, feature zipping, semantic alignment, or coefficient selection, but the review notes limits in scale or shared structure.The paper tests whether dimensional adaptation and direct interpolation can work across genuinely different LLM scales without those mechanisms.Source: §2 · p. 4

3.1 Problem Setup

• The setup considers pretrained models A and B, with A no larger than B in parameter dimensionality, and evaluates whether merging preserves capability or fails.It denotes their parameter vectors as θA and θB and the evaluation distribution as D.Source: §3.1 · p. 4

3.2 Union-Style Merging (Expand Then Merge)

• Union-style merging expands model A into model B’s space by copying compatible tensors and initializing new dimensions to limit functional perturbation.New linear parameters are zero-initialized, normalization scales use neutral values, and additional Transformer blocks begin as residual identity mappings.Source: §3.2 · pp. 4–5

3.3 Intersection-Style Merging (Truncate Then Merge)

• Intersection-style merging projects model B into model A’s space by copying compatible tensors, slicing oversized tensors, and removing excess layers.Because truncation discards parameters, the truncated branch is kept as a small perturbation.Source: §3.3 · p. 5

3.4 Evaluation Protocol

• Union and intersection use separate ratio grids, selecting the best checkpoint by downstream validation performance to test transfer near each anchor and locate collapse.The ratio grids are specified in the experiment section.Source: §3.4 · p. 5

• Qwen2.5 and Qwen3 model pairs ranging from 3B to 32B parameters are evaluated.Source: §4 · p. 5
• The benchmark suite covers mathematical reasoning, code generation, instruction following, NLU, commonsense, knowledge, and comprehensive reasoning.Source: §4 · p. 5, Table 1 · p. 5
• Union ratios control the larger checkpoint’s contribution; intersection ratios control the truncated larger checkpoint’s contribution.Source: §4 · p. 6
• The benchmark families and model-pair configurations are summarized in Tables 1 and 2.Source: Table 1 · p. 5, Table 2 · p. 5, §4 · p. 6

5 Main Results

• Across three union settings, direct averaging after deterministic expansion improves average scores over both sources; small-coefficient intersection merging also yields gains.Source: §5 · pp. 6–7

5.1 Knowledge-Structure Compatibility and Complementarity

• Partially distinct profiles support complementarity across categories; intersecting, non-nested profiles gain about four percentage points, whereas nested profiles change accuracy marginally.Source: §5.1 · p. 7

5.2 Scale Gap Drives Intersection Gains

• 0.5459 to 0.5716 for Qwen2.5-3B and 0.7045 to 0.7161 for Qwen2.5-14B; larger scale gaps yield stronger gains if truncation remains a small perturbation.Source: §5.2 · pp. 7–8, Table 7 · p. 7

6.1 Analysis of Gain Patterns

• Gains concentrate when source models have partially distinct competence profiles, with transferable complementary regions also surviving intersection-style truncation.The effect is more pronounced when the two models differ more substantially in scale.Source: §6.1 · p. 8

6.2 Ratio Sensitivity and Effective Region

• Union-style merging has two endpoint neighborhoods, whereas intersection-style merging is one-sided because truncation is lossy.The passage relates this ratio sensitivity to weight-space connectivity and alignment effects.Source: §6.2 · p. 8

6.3 Seesaw Effect

• 0.7525 for the best Qwen2.5-14B/32B merge exceeds both source averages, but the worst average falls to 0.7036 with task-level drops.Aggregate gains therefore coexist with capability-specific regressions.Source: §6.3 · p. 8

6.4 Perturbation Controls and Baseline Comparisons

• On RTE, the merge scores 0.7256 versus 0.6426 for baseline, 0.6534 for target-only scaling, and 0.6570 for matched noise.On GSM8K, matched noise scores 0.6808 and the merge scores 0.7028; the reported results indicate that comparable non-checkpoint perturbations do not fully explain the improvement.Source: §6.4 · p. 8
• Truncated-0.02 weighted averaging scores 0.5551, close to TIES at 0.5587 and above SLERP, the 3B baseline, and DARE on the reported average.The comparison tests competitiveness, not dominance over all merging algorithms.Source: §6.4 · p. 8

7 Conclusion

• Deterministic dimensional adaptation enables direct weighted averaging across heterogeneous LLM scales.The paper presents this as a competitive zero-training baseline for probing heterogeneous LLM merging.Source: §7 · p. 8
• Small-ratio union and intersection merging can preserve source behavior and transfer complementary capabilities across Qwen-family model pairs.The procedure uses no training, adapters, routing, or semantic alignment.Source: §7 · p. 8
• Compatibility remains local to the source pair, merging direction, and evaluation task, so task-level validation is essential before treating a merged checkpoint as uniformly improved.Source: §7 · p. 8

Limitations

• The evaluation focuses on Qwen-family checkpoints, offline benchmarks, and representative merging ratios.Source: Limitations · p. 9
• Long-context, multilingual, and tool-use settings remain outside the study’s evaluation.Source: Limitations · p. 9
• Efficiency, calibration, robustness, and safety settings also remain future evaluation areas.Source: Limitations · p. 9

Ethical Considerations

• Merged models may inherit unsafe behavior from source checkpoints, making safety checks necessary before deployment.Source: Ethical Considerations · p. 9
• Source-checkpoint biases may also persist in merged models.Source: Ethical Considerations · p. 9
• The passage identifies inherited unsafe or biased behavior as a deployment concern.Source: Ethical Considerations · p. 9

A Additional Task-Level Results

• Tables 10–12 catalog task-level best and worst outcomes for three Qwen model pairs under union-style merging.The captions identify Qwen2.5-14B/Qwen2.5-32B, Qwen3-4B-Thinking-2507/Qwen3-8B, and Qwen3-14B/Qwen3-32B; they do not provide outcome values here.Source: Table 10 · p. 12, Table 11 · p. 12, Table 12 · p. 12
• Tables 13–16 catalog task-level best and worst outcomes for four Qwen models under small-ratio intersection-style merging.The captions cover Qwen2.5-3B, Qwen2.5-14B, Qwen3-14B, and Qwen3-4B-Thinking-2507; they do not provide outcome values here.Source: Table 13 · p. 13, Table 14 · p. 13, Table 15 · p. 13

B Dimensional Adaptation Details

• EXPAND maps a smaller checkpoint into a larger architecture, while TRUNCATE projects a larger checkpoint into a smaller architecture; both are deterministic tensor maps applied before interpolation.They are not learned alignment modules.Source: B Dimensional Adaptation Details · p. 14

B.1 Union-Style EXPAND

• EXPAND uses deterministic slot insertion, zero-filled new branches, and residual identity initialization for added layers.Inserted-layer branch weights are zero and normalization scales are set to one.Source: Table 17 · p. 15, §B.1 · p. 15

B.2 Intersection-Style TRUNCATE

• TRUNCATE copies overlapping tensor prefixes and removes dimensions and layers outside the smaller architecture; the projection is intentionally lossy.It retains target-width MLP slices and follows the smaller model’s tied-embedding convention.Source: Table 18 · p. 16, §B.2 · p. 16
• The stability account is local, not a guarantee: nonlinear effects can destabilize middle ratios, and direct averaging assumes architecture-compatible models.The appendix treats collapse at µ ∈ (0.1, 0.9) as a diagnostic of the compatibility boundary.Source: §B.2 · p. 16

C.1 Ablation Controls and Baseline Comparisons

• The Qwen2.5-3B intersection comparison tests target-only scaling and matched-noise controls against weighted averaging after truncation.The controls assess whether gains could be explained by non-checkpoint perturbations of comparable scale.Source: Table 19 · p. 17, §C.1 · p. 17
• After truncation, weighted averaging is described as a competitive zero-training, zero-alignment baseline, not as dominant over all merging rules.The comparison uses representative merging rules evaluated in the same small-model parameter space.Source: Table 20 · p. 17, §C.1 · p. 17

C.2 GSM8K Ratio Sensitivity

• Tables 21 and 22 present GSM8K ratio sensitivity for union-style and intersection-style merging, respectively.The captions identify the benchmark and merging styles but provide no score values here.Source: Table 21 · p. 17, Table 22 · p. 17