PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Merging Scaling Laws in Large Language Models

Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, Hongxia Yang

arXiv:2509.24244v4cs.AI

TL;DR

Language-model merging lacks quantitative rules for predicting returns as model size and expert count change. This paper introduces and validates a floor-plus-tail power law, finding predictable diminishing returns as more experts are merged across sizes, domains, architectures, and methods.

  • Problem

    Language-model merging lacks an analogous quantitative account for anticipating convergence, comparing base sizes, and making budget-aware design choices.

  • Method

    The paper develops a compact floor-plus-tail power law linking base-model size N and merged-expert count k, supported by a theory of inverse-k gains.

  • Results

    R2 > 0.98 over all fitted points, with the same steep-then-flattening pattern across 10,866 models, sizes, domains, architectures, and four merging methods.

  • Takeaways & Limitations

    A lightweight fit can forecast loss-versus-k curves and guide expert-count and model-size tradeoffs for budget-aware merging decisions.

  • Takeaways & Limitations

    The law targets expected token-level cross-entropy and is not claimed to quantitatively predict every downstream benchmark curve, which can plateau earlier.

Abstract

from arXiv · show

We study empirical scaling laws for language model merging measured by cross-entropy. Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we add experts or scale the model size. We identify a compact power law that links model size and expert number: the size-dependent floor decreases with model capacity, while the merging tail exhibits clear diminishing returns in the number of experts. The law holds in-domain and cross-domain, tightly fits measured curves across diverse architectures and methods (Average, TA, TIES, DARE), and explains two robust regularities: most gains arrive early, and variability shrinks as more experts are included. Building on this, we present a simple theory that explains why gains fall roughly as 1/k and links the floor and tail to properties of the base model and the diversity across domains. This law enables predictive planning: estimate how many experts are needed to reach a target loss, decide when to stop adding experts, and trade off scaling the base model versus adding experts under a fixed budget--turning merging from heuristic practice into a computationally efficient, planable alternative to multitask training. This suggests a scaling principle for distributed generative AI: predictable gains can be achieved by composing specialists, offering a complementary path toward AGI-level systems.

1. Introduction

The paper introduces a predictive floor+tail scaling law linking base-model size and merged-expert count, addressing merging’s lack of quantitative planning guidance. Experiments across 10,866 models show consistent diminishing returns, shrinking variability, and convergence across methods and settings.

  • Motivation: The study addresses merging’s lack of a quantitative account for anticipating convergence, comparing rules across base sizes, and making budget-aware design choices.
  • Scaling law: The proposed compact scaling law couples model size N with merged-expert count k through a size-dependent floor and a diminishing merging tail.Larger models lower the floor and shrink tail amplitude, while adding experts produces steep early gains that taper as 1/(k+b).
  • Validation: 10,866 merged models spanning 0.5B to 72B, nine domains, and four methods validate the power law across diverse experimental settings.The methods are Average, Task Arithmetic (TA), TIES, and DARE.
  • Efficiency: Merging approaches multitask SFT performance while using negligible GPU-hours, and method gaps compress as expert count and model size grow.The comparison uses normalized loss and GPU-hours.
  • Empirical regularities: The study identifies robust regularities: larger models are easier to merge, most gains arrive early, mixed domains improve pooled generalization, and method means and variability converge at scale.
  • Generality: Order sensitivity fades as expert count grows, while the power-law shape transfers across backbones and applies in both in-domain and cross-domain settings.

2. Background, Related Work, and Setup

This section defines model merging through task-vector composition and frames the study around base-model size and expert count as its explicit scaling axes. It introduces four merging methods and the controlled/open-source expert, data, and cross-entropy evaluation setups used to study them.

  • Notation: Task vectors are parameter differences from the base model, optionally restricted to adapter or LoRA subspaces, and are merged with equal weights unless otherwise stated.The notation uses N for base-model size, M for the expert set, k for the number of merged experts, and θ0 for the base model.
  • Model Merging: Model merging integrates independently trained models by aggregating parameters in full parameter spaces or modular adapter and LoRA subspaces.The common formulation manipulates task vectors, which represent changes relative to a base pretrained model.
  • Expert capacity: Base-model size N and expert count k are the explicit scaling axes, while expert-training capacity is held fixed in controlled Qwen experiments and represented through effective-update distributions.Changing LoRA rank, adapter width, fine-tuning token budget, or expert quality would shift the fitted floor, tail amplitude, and related update statistics.
  • Merging k Experts: The study evaluates Average, TA, TIES, and DARE, treating preprocessing such as trimming, masking, sparsification, or rescaling as transformations of effective updates.Average and TA use unmodified task vectors, while TIES and DARE apply rule-specific preprocessing before equal-normalized composition.
  • Evaluation: Evaluation reports token-level cross-entropy by scoring 30M held-out tokens per domain and averaging loss across expert subsets.For N>8B, a uniform random subset is used to control cost.

3. Scaling Laws with Merging Experts and Model Size

Across in-domain and cross-domain evaluations, expected merge loss follows a size-dependent floor plus a diminishing-returns expert tail. Larger models lower the floor and shorten the tail, while merging more experts improves accuracy and reduces variability, with most gains arriving early.

  • Empirical scaling law: The same scaling law holds in-domain and cross-domain, including macro-averaged evaluation over nine heterogeneous domains.Cross-domain gains are monotone, steep early, and flatten into a 1/(k+b) tail.
  • Empirical scaling law: Expected cross-entropy decreases smoothly with expert count k and follows a floor-plus-tail law, with diminishing returns as k increases.The tail is modeled as A(N)/(k+b), while L∞(N) is the limiting loss as k→∞.
  • Model-size scaling: Larger models lower the loss at fixed k and require fewer experts to approach the floor.At k=9, domain-averaged CE drops from 0.739 (@0.5B) to 0.430 (@32B), a 41.9% reduction.
  • Stability: Subset-loss variability decreases as more experts are merged, with generic standard deviation decaying as 1/k and variance contracting as Θ(1/k2).Large-k merges therefore show better mean performance and reduced run-to-run variability.
  • Expert-count scaling: Most improvement arrives early: the median curve reaches 85% by k=5 and 90% by k=6, with k90 concentrated in {5, 6}.The marginal gain decays roughly as k−2, producing a clear elbow beyond the first few experts.

4. Further Analysis and Recipe

Further analyses show that the unified floor-plus-tail law remains stable across candidate-pool sizes and backbones, while early measurements can forecast expert-count curves. They also confirm rapidly diminishing order sensitivity and caution that downstream metrics plateau qualitatively but need not follow the cross-entropy law quantitatively.

  • Candidate-pool size: Reducing the candidate pool from M=8 to M=7 leaves floors stable but changes tail behavior mainly on science-like domains.A larger, more diverse pool supplies complementary donors and reduces residual cross-domain mismatch.
  • Early forecasting: Three points at k∈{1, 2, 4} closely forecast the full trajectory, with implied k⋆ concentrating around 5∼6.The floor anchors the late regime while the tail controls the early drop, making both parameters identifiable without overfitting.
  • Order sensitivity: About 83% shrinkage in whisker length by k=8 demonstrates that donor-order effects fade rapidly as experts accumulate.At k=9, relative range reduction is ≈24% for 0.5B, 32% for 32B, and 34% for 72B.
  • Cross-backbone validation: R2=0.999 for LLaMA-3.2 3B and R2=0.995 for LLaMA-3 8B show that both backbones follow the same inverse-tail law.Both models obtain roughly 80% of total improvement with only k≈4−5 experts.
  • Downstream behavior: Downstream metrics generally improve with k and plateau, but they can saturate earlier than token-level CE and are not claimed to obey the same quantitative law.Task Arithmetic improves quickly from k=1 to k≈3, while TA and TIES on 8B share the same qualitative trajectory for k≥2.

5. Conclusion

The paper presents a predictive floor+tail power law linking model size with the number of merged experts. The law captures recurring empirical patterns and holds across domains and architectures.

  • Conclusion: The floor+tail power law links model size and merged-expert count in a simple, predictive scaling law.It provides a unified form for model merging behavior.
  • Conclusion: Larger bases lower the size-dependent floor, while most improvement arrives at small k and variance contracts as experts are added.The same law also captures compressed method gaps and the diminishing importance of merge order at scale.
  • Conclusion: The power-law form holds both in-domain and cross-domain and transfers across architectures.This supports its applicability across different evaluation settings and model designs.

Impact Statement

The work advances understanding of model merging through a principled scaling law relating performance to model size and merged-expert count, with theoretical and practical guidance for efficient expert merging.

  • Impact Statement: The proposed scaling law characterizes how model-merging performance evolves with model size and the number of merged experts.It is intended to provide both theoretical insight and practical guidance for efficient expert merging.
  • Impact Statement: The techniques operate on trained models without introducing new learning objectives or data sources.This may help reduce unnecessary computation and resource usage in large-scale model development.

Reproducibility statement

The paper supports reproducibility by using publicly available models and datasets, documenting methodological and implementation details, and releasing complete source code for replication and verification.

  • Reproducibility statement: Publicly available models and datasets underpin the reported results.The authors state that all models and datasets used in the work are publicly available.
  • Reproducibility statement: Section 2 details methodological choices, data sources, and evaluation protocols, while Appendix D documents implementation details and hyperparameters.These materials provide the methodological and configuration information needed to reproduce the experiments.
  • Reproducibility statement: Complete source code is provided as supplementary material to facilitate replication and independent verification.The release is intended to enable others to replicate and independently check the reported results.

Limitations. · A. Model Merging Recipes

The paper’s main scaling claim applies to expected token-level cross-entropy under merging, while downstream benchmark scores offer complementary utility evidence but may follow different behavior. A unified form represents the merging recipes in Table 1.

  • Limitations.: The main claim targets expected token-level cross-entropy under merging.Cross-entropy is the quantity used for the primary scaling analysis.
  • Limitations.: Cross-entropy is dense and relatively low-variance.These properties motivate its use as the main measurement.
  • Limitations.: Cross-entropy directly aligns with the local second-order analysis behind the floor+tail form.The measurement choice is tied to the analysis that produces the proposed scaling form.
  • Limitations.: Downstream benchmark scores provide complementary evidence of utility but need not obey the same scaling law.Benchmark behavior is therefore not assumed to match cross-entropy scaling exactly.
  • Limitations.: Benchmark scores can plateau earlier than cross-entropy as k grows.They are sparser, more thresholded, and typically noisier aggregates of task success.
  • A. Model Merging Recipes: A unified form represents all merging recipes in Table 1.The passage states that the recipes are handled together using one form.

B. Detailed proof of Theorem 3.1

For fixed model size, the proof derives a 1/k merging tail by Taylor expansion and moment control under smoothness, task-vector, and equal-weight assumptions. It then relates the theoretical coefficients to the fitted intercept and tail amplitude, with approximation errors absorbed into the model’s N-dependent constants.

  • Assumptions: Under Assumption 3.2, the proof assumes a locally twice continuously differentiable loss with M-Lipschitz Hessian, i.i.d. task vectors with finite sixth moment, and α_i,k = c/k normalization.Expectations use a uniform random k-subset, with the same orders under sampling with replacement.
  • Moment control: The mean-corrected merging step has zero expectation, covariance c^2Σ/k, and a third-moment remainder of order O(k^-3/2).The covariance follows from averaging centered task vectors, while the third-moment rate uses the finite sixth-moment condition.
  • Taylor expansion: Taylor expansion at θ0 + cµ shows that the averaged loss approaches its asymptote with a 1/k tail.The tail coefficient is determined by the gradient, Hessian, task-vector covariance, and normalization at the expansion point.
  • Theorem conclusion: Combining the expansions yields the fixed-N form fitted in the main text: an N-dependent intercept plus a diminishing 1/k tail with a quantitative remainder.The N-dependent constants absorb small base-point and curvature-surrogate discrepancies into L∞(N) and A(N).

C. Detailed proof of Corollary 3.2 … H. In-Domain Fits

The appendix proves the asymptotic variance behavior underlying Corollary 3.2 and describes how merging experiments evaluate, sample, and model expected loss across expert subsets. It also characterizes expert-training scaling with model size and computation, while empirical constructions focus on smooth mean-loss curves as subset variability narrows.

  • C. Detailed proof of Corollary 3.2: For α > 0, the proof establishes constants C1, C2 > 0 and k0 yielding the stated asymptotic bound for all k ≥ k0.The remainder contributes a one-sided O(1/k2) upper bound.
  • C. Detailed proof of Corollary 3.2: When a⊤Σa = 0, the linear contribution vanishes, leaving the uniform bound derived from the quadratic and remainder terms.This is the degenerate-linear-term case.
  • C. Detailed proof of Corollary 3.2: The quadratic fluctuation bound is tight when HS is nonzero on the range of Σ and fourth central moments are non-degenerate there.The passage describes this as a mild condition satisfied in the experiments.
  • D. Expert Model Details: Evaluation uses token-level cross-entropy, computed as average negative log-likelihood over 30M randomly sampled validation tokens per domain.Sampling addresses variation across repeated runs and execution environments.
  • D. Expert Model Details: Because merged loss depends on the selected domains as well as k, the experiments enumerate possible expert subsets and compute expected loss over them.For larger models, computational overhead motivates sampling rather than complete enumeration.
  • E. Sampling Algorithm: The sampling algorithm starts with a base sequence and its reverse, then greedily adds random candidates maximizing minimum Hamming distance from existing permutations.It generates 1000 random candidates at each subsequent iteration.
  • E. Sampling Algorithm: With k = 15 on the 0.5B model, sampled merging curves closely align with full-combination curves in overall trend and numerical values.This supports using diverse permutation sampling to reduce combination costs.
  • F. Scaling Laws for Expert Model Training: Expert-model performance generally improves with model size and computational budget, while domain-specific performance curves can differ.The analysis also varies training-token count and overall computation across domains.

H.0.1. IN-DOMAIN (SINGLE-DOMAIN EVIDENCE) … J. Core Questions

Across in-domain and cross-domain evaluations, merging cross-entropy follows a size-dependent floor plus a diminishing-return tail in the number of experts. The evidence shows steep early gains, improved stability with larger models, and shrinking method gaps as more experts are added.

  • H.0.1. IN-DOMAIN (SINGLE-DOMAIN EVIDENCE): In-domain CE decreases near-monotonically with k and follows a 1/(k+b) tail, with most gains arriving by k≈5.At 0.5B, macro in-domain CE drops from ≈0.816 at k=1 to ≈0.739 at k=9; at 32B, it drops from ≈0.493 to ≈0.430.
  • H.0.1. IN-DOMAIN (SINGLE-DOMAIN EVIDENCE): At fixed k=9, macro CE moves from ≈0.739 (@0.5B) to ≈0.430 (@32B), about −42%, while both the floor L∞(N) and tail amplitude A(N) shrink with N.Per-domain joint fits for Average give exponents β ∈[0.33, 0.42] and high R2.
  • H.0.1. IN-DOMAIN (SINGLE-DOMAIN EVIDENCE): At k=9, the median in-domain CE is forecast to drop another ∼6–10% from 32B to 72B.Full per-domain parameters for Average/TA/TIES, including b, and the 72B forecasts are reported in Appendix H.
  • H.1. Mean CE: Joint (N, k) Fits: The joint mean-CE law fits per-domain parameters for Average, TA, and TIES using weighted nonlinear least squares, with weights proportional to k and R2 computed on held-in k grid points.The law uses L∞,d(N) = L∗,d + BdN −βd, Ad(N) = A0,dN −γd, and finite-k offset b0,d.
  • H.2. Variance: Joint (N, k) Fits by Method: Variance predictions are reported for k ∈{1, 3, 5, 9} across Average, TA, and TIES fit-parameter and N=72B tables.The cited tables provide variance fit parameters and variance at N=72B for each method.
  • I.0.1. CROSS-DOMAIN (POOLED EVIDENCE): Cross-domain macro-averaged CE follows the same L∞(N) + A(N)/(k+b) law, with monotone curves, steep early gains, and a short inverse tail.TA and TIES(0.5) show slightly faster early drops, gaps narrow with k, and TIES(1) has a small bounded non-monotonicity at 3B captured in the fit.
  • I.1. Variance Behavior (Both Settings): Variance exhibits a near-1/k decline, decreases with larger N at fixed k, and typically orders methods as TIES < TA < Average at small k, with gaps vanishing as k grows.Chemistry @0.5B, Average declines from 0.0385→0.00108 by k=8; physics, Average, k=1 declines from 0.0239→0.0128 from 0.5B to 32B.

J.1. Per-domain fits, kε examples, and robustness … K. Do Downstream Metrics Follow the Same Trend?

Across domains and methods, larger base models lower the merging floor while expert-count gains arrive early and then diminish. These patterns are robust, make kε a practical saturation budget, and show that method differences and variability narrow as more experts are merged.

  • J.1. Per-domain fits, kε examples, and robustness: At k=9, macro CE falls from 0.739 at 0.5B to 0.430 at 32B, a 41.9% decrease consistent with a lower floor and weakly shrinking tail.This supports model-size scaling as the dominant driver of improvement at high expert count.
  • J.1. Per-domain fits, kε examples, and robustness: Code reaches ε=0.01 saturation at kε=8 for 0.5B and kε=5 for 32B, while biology remains near kε≈18 because its tail is nearly flat.For code, A(0.5B)≈0.074 and A(32B)≈0.046; biology has A(0.5B)≈0.173 and A(32B)≈0.177.
  • J.1. Per-domain fits, kε examples, and robustness: Changing weighting schemes or censoring tiny high-k points barely changes floor exponents, so extrapolation should treat floors as the dominant N-driver and tails as weakly decreasing or flat.The resulting kε provides a practical experts-to-saturation budget.
  • J.2. Most of the gain comes from the first few experts: The median fractional return reaches 85% by k=5 and 90% by k=6, showing that most merging improvement arrives among the first few experts.The k90 heatmap concentrates at k∈{5,6}, or roughly half to two-thirds of a 9-expert pool.
  • J.3. Additional plots, tables, and details: Across Average, TA, TIES, and DARE, pooled improvement is mostly realized by k≤6, method differences narrow with k, and increasing N lowers both the pooled floor and tail.Per-method fits use L(N,k)=L∞(N)+A(N)/(k+b), with an extra bounded term only for strongly nonlinear TIES curves.
  • Extended evidence: At k=2, TA and TIES(0.5) are typically 1%∼2% better than Average, while by k=9 TIES(0.5) usually retains a ≈1%∼2% edge and TA is near-tied or slightly worse.The reported signed gaps show TA at +0.7% or +1.4% worse and TIES(0.5) at −1.1% or −2.2% better in the listed comparisons.
  • Extended evidence: At N=32B and k=2, across-merge variance is 9.67×10−4 for Average, 7.83×10−4 for TA, and 6.50×10−4 for TIES 0.5, converging to about (3−4)×10−5 by k=8.Relative to Average, TA is −19% and TIES 0.5 is −33% at k=2.
  • Extended evidence: A small TIES(λ=1) non-monotonicity at 3B is captured by a bounded D k/(k+q) term, whereas λ=0.5 restores the standard monotone 1/(k+b) tail.Together, these results support method differences as second-order effects that shrink quickly with k.

K.1. Overall Results … M.4. Additional Model-Size Slices for Method Comparison

Across backbones, merge methods, and domain pools, downstream quality and cross-entropy improve as more experts are merged, with diminishing returns and increasing stability. Detailed analyses attribute these trends to a shared 1/(k+b) tail, structured donor–receiver synergies, and reduced order sensitivity.

  • K.1. Overall Results: Downstream accuracy generally rises with expert count before saturating, with the plateau often arriving earlier than for cross-entropy.LLaMA-3.1 8B with Task Arithmetic increases from 0.41 at k=1 to 0.47 at k=5, with rapidly diminishing gains after k≈3.
  • K.2. Detailed Cases: The detailed evaluation merges a common base model with five specialists spanning math, code, multilingual understanding, safety, and instruction-following across three backbones and two merge rules.The expert subsets are evaluated for k∈{1, . . . , 5}, metrics are normalized so larger is better, and mean accuracy is averaged across tasks and subsets at fixed k.
  • M. Cross-Domain Synergy: DARE synergy matrices show structured interactions: within-science and within-math pairs are positive, whereas cross-block interactions are weakly negative, with code→(discrete, geometry) mildly positive.At 32B, the strongest positive off-diagonal pairs include biology→chemistry (+0.076), while algebra→physics is the largest listed negative (−0.026).
  • L.1. Fitted Scaling-Law Parameters on LLaMA Backbones: Fitted LLaMA-backbone curves achieve near-unity R2 with a shared 1/(k+b) tail, supporting the same diminishing-return form across replicated cross-backbone series.The fits use L(k) = L∞+ A k+b and report small residuals and endpoint values L(k=1)/L(k=9).
  • M.1. Details under DARE: Across DARE model sizes, block-level synergies vary in magnitude but preserve the distinction between positive within-block effects and mostly negative cross-block effects.Reported block means include, at 32B, SM→M=0.012, SS→S=0.073, SM→S=−0.013, and SS→M=−0.005.
  • M.2. Details for Order sensitivity and 1/(k+b) fit: Order sensitivity collapses as experts increase: from k=1 to k=8, standard deviation drops ∼79%−81% and range drops ∼83% across 0.5B, 32B, and 72B.A shared small offset ˆb≈2 with c0≈0 explains most of the fitted decay, while c1 shrinks up to mid-scale.
  • M.4. Additional Model-Size Slices for Method Comparison: Additional model-size slices compare mean CE loss across different expert counts, extending the paper’s method-comparison analysis across model capacity.Figure 22 reports mean CE Loss versus model size with different k.
Loading 2509.24244v4…