SLIM: Simplex-Lattice Interpolation Merging

1 INTRODUCTION

• Generation-based benchmark evaluation makes coefficient searches costly, motivating the question of how few target-metric measurements can support competitive merge selection.The target metric here is AVG, the unweighted mean of domain scores on a common scale.Source: §1 · p. 1
• Restricting coefficients to the simplex reduces the independent parameters of a general quadratic from (T + 1)(T + 2)/2 to T(T + 1)/2.Simplex coefficients are nonnegative and sum to one; the identification minimum alone does not guarantee stable fitting or accurate prediction.Source: §1 · p. 1
• The 36 construction evaluations support accurate prediction of aggregate AVG at unseen mixtures of all eight experts on both architectures.Figure 1 tests 150 held-out interior mixtures; domain-wise prediction fidelity varies.Source: Figure 1 · p. 2, §1 · p. 2
• SLIM applies Scheffé’s {T, 2} simplex-lattice design, using individual experts and equal-weight pairs to fit a quadratic and optimize it without further target-metric evaluations.The design uses exactly T(T + 1)/2 measurements and numerical optimization with the surrogate’s analytic gradient.Source: §1 · p. 2
• With 36 evaluations, SLIM exceeds MAP at 45 in all four architecture-by-distribution settings and matches evolutionary merging using 150–200 evaluations in three settings.On Llama ID, methods searching the broader coefficient domain achieve higher performance.Source: §1 · p. 2

Model Merging

• Prior coefficient-selection approaches range from data-free rules and entropy minimization to searches guided by measured downstream performance; candidate LLM merges can be costly to evaluate.MERGE3 combines evolutionary merging with dataset reduction and item-response-theory performance estimation.Source: §2 · pp. 2–3

Surrogate-Based Coefficient Optimization

• Surrogate-based methods use measured merge performance to guide coefficient or configuration search, including Gaussian-process, random-forest, and quadratic-surrogate approaches.MAP fits quadratic surrogates to randomly sampled coefficient vectors; Bayesian MAP and Nested MAP reduce evaluation costs through adaptive sampling and hierarchical merging.Source: §2 · p. 3
• SLIM uses a fixed simplex-lattice design to interpolate AVG from T(T + 1)/2 measurements, the minimum needed to identify a general quadratic on the simplex.The paper tests whether this design predicts unseen multi-expert mixtures and supports competitive coefficient selection.Source: §2 · p. 3

Mixture Experiments

• Mixture experiments model responses to component proportions that are nonnegative and sum to one, making their domain a simplex.For LLM merging, experts are components, coefficients are proportions, and AVG is the response; Scheffé’s design uses vertices and edge midpoints to determine a quadratic.Source: §2 · p. 3

3.1 PROBLEM SETTING

• For experts fine-tuned from a common base, linear task-vector merging defines θ(λ), and coefficient selection maximizes target metric M(λ) over feasible coefficients.Source: §3.1 · p. 3

3.2 QUADRATIC SURROGATE AND EVALUATION DESIGN

• On the simplex, a general quadratic has T(T + 1)/2 independent coefficients; SLIM measures the T vertices and edge midpoints to identify them.Source: §3.2 · p. 4

3.3 QUADRATIC INTERPOLATION

• The quadratic interpolant reproduces all measured design-point values and is unique, while accuracy at unseen mixtures must be assessed empirically.Source: §3.3 · p. 4

3.4 COEFFICIENT SELECTION

• A constrained numerical optimizer maximizes the surrogate without further target-metric evaluations, but the returned solution is not guaranteed globally optimal because the surrogate may be non-concave.Source: §3.4 · p. 4

4.1 EXPERIMENTAL SETUP

• Experiments use Gemma-2-9B and Llama-3.1-8B with eight experts across five domains; ID benchmarks guide selection while OOD benchmarks are held out.Source: §4.1 · p. 5

4.2 SURROGATE FIDELITY

• On 150 held-out mixtures involving all eight experts, aggregate AVG Spearman ρ is 0.936 on Gemma and 0.959 on Llama, with MAE 1.93 and 1.07 points, respectively.Edge-point MAE is 1.74 on Gemma and 1.38 on Llama; domain fidelity varies, including Gemma math ρ = −0.24.Source: §4.2 · p. 6

4.3 MERGE PERFORMANCE AND EVALUATION COST

• With 36 evaluations, SLIM costs 5.2 GPU-hours on Gemma-2-9B and 3.9 on Llama-3.1-8B, beats weight averaging in all four settings, and exceeds MetaGPT in three.On Gemma OOD, MetaGPT scores 50.6 versus SLIM’s 50.3.Source: §4.3 · p. 6
• 36-evaluation SLIM outperforms MAP at 45 evaluations in all four settings; Figure 3 plots ID and held-out OOD AVG against evaluation budget.Source: §4.3 · p. 6, Figure 3 · p. 7
• At 30 and 40 evaluations, evolutionary merging scores below SLIM on Gemma ID/OOD and Llama OOD but exceeds it on Llama ID; at 150–200 evaluations, SLIM is comparable or higher depending on setting.Source: §4.3 · pp. 6–7

4.4 SELECTED MERGING COEFFICIENTS

• SLIM concentrates weight on instruction and safety experts, while MAP and evolutionary merging use coefficient sums above one and can attain higher ID AVG than SLIM.MAP sums range from 3.31 to 3.75 and evolutionary-merging sums from 2.50 to 2.83, compared with one for SLIM.Source: §4.4 · p. 7

5.1 EFFECT OF EVALUATION-POINT PLACEMENT

• The placement comparison draws random interior points without replacement, fits the same quadratic by least squares or ridge, and evaluates predictions on held-out points.The procedure repeats 1,000 times, while the 36-point lattice surrogate remains fixed.Source: §5.1 · p. 7, Table 2 · p. 8
• On Llama, MAP and evolutionary merging choose coefficient sums above one, while SLIM’s sum is one; both alternatives achieve higher ID AVG at these budgets.SLIM concentrates weight on instruction and safety experts on both architectures.Source: Figure 4 · p. 8
• At 36 evaluations, the lattice has higher mean rank correlation and lower mean MAE than random LS/ridge fits on both architectures; MAE is ~1.93 for Gemma and ~1.07 for Llama, versus ridge’s 2.90 and 2.24.The lattice’s win rate first falls below 50% at n = 60 for Gemma and n = 105 for Llama.Source: §5.1 · pp. 7–8, Table 2 · p. 8

5.2 EFFECT OF SURROGATE OPTIMIZATION

• Adding equal-weight expert pairs accounts for most of the ID gain over vertex-only selection; surrogate optimization adds a smaller ID gain, with OOD AVG decreasing on Gemma and increasing on Llama.Table 3 contrasts selection among eight vertices or 36 lattice points with SLIM’s surrogate optimization.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 EFFECT OF POLYNOMIAL DEGREE

• The 120-point cubic extension yields small ID gains but lower OOD AVG on both architectures, so increased degree and evaluation budget do not consistently improve merge performance.Table 3 compares the cubic extension with quadratic SLIM under simplex-constrained optimization.Source: §5.3 · pp. 8–9, Table 3 · p. 9

Quadratic Approximation and Model Degree

• Quadratic surrogates predict AVG at unseen mixtures of all eight experts on both architectures, but domain-wise fidelity varies; cubic models use 120 rather than 36 evaluations and yield small ID gains but lower OOD AVG on both.When greater surrogate expressiveness improves generalization remains an open question.Source: §6 · p. 9

Evaluation Design and Sample Efficiency

• At 36 evaluations, lattice points yield higher mean rank correlation and lower MAE than random designs, including ridge fits; T(T + 1)/2 identifies a general simplex quadratic, not necessarily a good merge.Evaluation placement matters even with the surrogate class and coefficient domain fixed.Source: §6 · p. 9

The Value of Surrogate Optimization

• Adding equal-weight pairs accounts for most ID improvement over the best individual expert; optimizing beyond the best measured lattice point gives small ID gains, lowers OOD AVG on Gemma, and raises it on Llama.The surrogate enables continuous coefficient selection without further target-metric evaluations, but its added performance benefit depends on the setting.Source: §6 · p. 9

The Role of the Simplex Constraint

• The simplex contains competitive merges that generalize beyond selection benchmarks, but Llama ID comparisons do not isolate coefficient-scale effects: MAP and evolutionary merging select higher-scoring candidates with sums above one.The comparisons motivate varying mixture proportions and overall task-vector scale while controlling surrogate-construction cost.Source: §6 · p. 9

7 CONCLUSION

• SLIM selects LLM merging coefficients with a quadratic surrogate built from individual experts and equal-weight pairs, requiring the minimum evaluations for quadratic identification on the simplex.Experiments on two architectures show accurate prediction at unseen mixtures and competitive merging under limited evaluation budgets; matched-budget comparisons highlight evaluation-point placement.Source: §7 · p. 9

A NOTATION

• Table 4 defines symbols for merging coefficients, the target metric, and surrogate construction..Source: Table 4 · p. 13

B.1 DERIVATION AND UNIQUENESS

• Vertex and edge-midpoint measurements determine the quadratic’s coefficients, and each evaluation supplies one linear constraint, so fewer evaluations cannot identify an unrestricted quadratic in this space.The derivation states that the vertex and midpoint values uniquely determine the coefficients.Source: §B.1 · p. 14

B.2 LINEARITY UNDER DOMAIN AGGREGATION

• For fixed domain weights, interpolating aggregated measurements gives the same surrogate as aggregating domain-wise interpolants; recording domain scores at the same design points requires no additional evaluations.AVG uses weights w_d = 1/D.Source: §B.2 · pp. 14–15

B.3 SLIM ALGORITHM

• SLIM constructs its design, evaluates each mixture, builds the quadratic surrogate, numerically maximizes it over the simplex, and returns the selected coefficients.The algorithm takes base parameters and task vectors as inputs.Source: §B.3 · p. 15

B.4 CUBIC EXTENSION FOR THE DEGREE ABLATION

• The cubic design uses simplex vertices, two unequal-weight points per expert pair, and the equal-weight mixture of each expert triple.Source: §B.4 · p. 15
• 120 evaluations are required for the cubic design when T = 8.Source: §B.4 · p. 15
• The cubic interpolant reproduces every measured design-point value and is unique among polynomials of degree at most three on the simplex.Source: §B.4 · p. 15
• The cubic interpolant is numerically maximized over the simplex, requiring no further model evaluations and retaining the 120-evaluation budget for T = 8.Source: §B.4 · p. 15

C.1 MODELS AND CHECKPOINTS

• Each Gemma-2-9B and Llama-3.1-8B expert set contains five domain-specific checkpoints and three additional checkpoints, including the official instruction-tuned model, Instruction B.Source: §C.1 · p. 16

• ID benchmarks are used for coefficient selection and reported ID results, while OOD benchmarks are held out from selection.Source: §C.2 · p. 16, Table 6 · p. 17
• ID translation evaluation uses 100 dev sentences per target language, versus all 1,012 devtest sentences per language for OOD, across eight languages.Source: §C.2 · p. 16
• The multilingual score is mean chrF2 across eight languages; WildGuardTest includes 749 evaluated examples after excluding five entries with missing values.Source: §C.2 · p. 16
• Merged candidates use the base model’s configuration and tokenizer, with raw prompts for lm-eval and FLORES+, an Instruct template for AlpacaEval, and shared chat formatting for safety.Source: §C.2 · p. 16

C.3 COEFFICIENT-SELECTION IMPLEMENTATIONS

• MAP fits nested samples of 45, 90, and 135 points, including eight vertices, and uses adjusted score scaling, parameter initialization, and search bounds.Source: §C.3 · p. 17
• Evolutionary merging uses one CMA-ES run of 20 generations per seed and reports the candidate with the highest measured ID AVG at each checkpoint.Source: §C.3 · p. 17
• SLIM optimizes its interpolant with analytic gradients and fixed simplex initializations, retaining the converged solution with the highest surrogate value without further target-metric evaluations.Source: §C.3 · pp. 17–18

C.4 RANDOM-DESIGN FITS

• Random-design least-squares and ridge fits use the same eight linear and 28 pairwise Scheffé-basis terms, with held-out points reserved for fidelity assessment.Source: §C.4 · p. 18

C.5 NUMERICAL RESULTS FOR FIGURE 3

• Table 7 reports ID and held-out OOD performance across coefficient-selection budgets, with coefficients selected using ID evaluations only.Source: §C.5 · p. 18

C.6 DOMAIN-WISE SURROGATE FIDELITY

• Table 8 assesses predictions on 150 held-out mixtures; its AVG row evaluates the aggregate objective, while domain-wise fidelity varies.Source: §C.6 · p. 18, Table 8 · p. 19

C.7 QUADRATIC INTERPOLATION ALONG SIMPLEX EDGES

• MAE is 1.74 on Gemma and 1.38 on Llama, while maximum absolute errors are 8.08 and 4.28, respectively, across held-out edge mixtures.Source: §C.7 · pp. 18–19, Figure 5 · p. 20, Figure 6 · p. 21