PaperlayerView exact revision on arXiv

Source-linked AI summary

Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, Anton Protopopov, Alexis Audran-Reiss, Michael Slater, Despoina Magka, Tatiana Shavrina, Roberta Raileanu, Yoram Bachrach

arXiv:2511.13254v2cs.CL

TL;DR

LLM training is resource-intensive, and prior souping approaches can collapse precision-sensitive capabilities. SoCE uses benchmark category correlations to select expert models and optimize their weights, improving performance and consistency while achieving state-of-the-art BFCL results.

  • Problem

    LLM training is resource- and time-intensive, while prior souping approaches can collapse precision-sensitive capabilities such as tool use.

    Source: §1 · pp. 1–2

  • Method

    SoCE selects expert models for weakly correlated benchmark categories and combines them using optimized weighted averaging.

    Source: §3 · pp. 4–5

  • Results

    80.68% BFCL accuracy, a 2.84% improvement over the previous best-performing individual model, xLAM-2-70b-fc-r at 78.45%; SoCE also improves performance consistency across categories.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    SoCE offers a low-cost opportunity to reuse existing pretrained models and may lead to significant computing-resource savings.

    Source: §5 · p. 9

    None of the considered benchmarks provides validation and test splits, so candidate selection and weight optimization relied on direct leaderboard scores.

    Source: General Method Limitations · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive computational resources and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. We observe that previous souping approaches can lead to collapse in precision-sensitive LLM capabilities. In this paper, we introduce SoCE, a principled approach for model souping to overcome this shortcoming. The proposed method utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous approaches, our method leverages the observation that different clusters (or categories) of points within a benchmark often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that SoCE improves performance and robustness across multiple domains and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

1 Introduction

Model souping offers a way to improve models without expensive retraining, but prior approaches can collapse precision-sensitive capabilities. SoCE uses benchmark category structure to select and weight expert models, reporting state-of-the-art BFCL performance and improved consistency across categories.

1 Introduction

  • LLM training remains resource- and time-intensive, motivating model souping as a way to improve performance without expensive retraining.Model souping averages weights from models of the same architecture.Source: §1 · p. 1
  • Previous souping approaches can collapse precision-sensitive capabilities, including tool use, long-context handling, and reasoning.The introduction cites Tables 3, 6, and 8 for these failures.Source: §1 · p. 2
  • SoCE selects expert models for weakly correlated benchmark categories and combines them with non-uniform weights.The method uses benchmark composition to guide candidate selection and weighting.Source: §1 · p. 2
  • SoCE achieves state-of-the-art BFCL results and outperforms existing baselines across diverse domains.The paper also reports improved performance consistency across benchmark categories.Source: §1 · p. 2
  • The paper also examines game theory and Shapley values, prior merging assumptions, cross-benchmark effects, and FLORES results.Its additional analyses include qualitative BFCL results and the effect of souping for one benchmark on others.Source: §1 · p. 2

2 Related Work

Prior work explores model souping and automatic merging to improve model performance and efficiency. SoCE differs by selecting and weighting models through correlations among categories within a single benchmark.

2 Related Work

  • Earlier model-souping work studied checkpoint averaging and proposed uniform, greedy, and learned weighting strategies, with greedy souping reported as the most promising.Weight averaging was linked to flatter, better-generalizing minima, and work also explored interpolation for robustness under distribution shift.Source: §2 · pp. 2–3
  • SoCE targets a single benchmark’s internal category structure, unlike prior souping focused mainly on combining expertise across benchmarks or domains.It uses Pearson correlations between subcategory scores to guide candidate selection.Source: §2 · p. 3
  • Automatic merging methods address parameter conflicts or learn merging weights, while SoCE uses public evaluation scores and preserves weights through weighted averaging.The paper summarizes method properties in Table 1 and refers readers to Table 7 for more methods.Source: Table 1 · p. 2, §2 · p. 3

3 Methodology: Soup Of Category Experts

SoCE uses correlations among benchmark categories to identify complementary model expertise. It selects expert models for weakly correlated categories, optimizes their weights, and averages their weights into a final model.

3 Methodology: Soup Of Category Experts

  • Benchmark category performance can be strongly correlated or weakly correlated across models, suggesting categories may reflect shared or distinct competencies.On BFCLv3, multi-turn categories correlate at 0.96–0.98, while Multi-turn-base and Live Accuracy correlate at 0.07.Source: §3 · p. 4
  • SoCE identifies weakly correlated category clusters and selects expert models for each, combining complementary expertise with optimized weighted averaging.The method is formalized as a four-step procedure: correlation analysis, expert selection, weight optimization, and model souping.Source: §3 · pp. 4–5
  • The method searches a uniform grid of model weights from 0.1 to 0.9 in 0.1 increments, with weights summing to one.Equal weighting is also tested as a uniform-souping comparison.Source: §3 · p. 5

4 Experiments

The experiments evaluate SoCE and souping baselines across function calling, multilingual math, and long-context benchmarks. They report BFCL state-of-the-art results, gains on other benchmarks, and more consistent category-level performance after souping.

4.1 Benchmarks and Baselines

  • Experiments cover BFCL function calling, MGSM multilingual math, and a three-category ∞Bench long-context subset, comparing individual models and several soup baselines.The design also compares merging methods on BFCL and isolates candidate-selection and weight-optimization effects.Source: §4.1 · pp. 5–6

4.2 Numerical Takeaways

  • 80.68% BFCL accuracy for the 70B SoCE soup beat the prior best individual model’s 78.45% by 2.84%; the 8B soup reached 76.50%, 5.7% above its size-class leader.The respective baselines were xLAM-2-70b-fc-r and xLAM-2-8b-fc-r.Source: §4.2 · pp. 6–7
  • TIES and DARE approaches scored 0% on BFCL Multi-Turn and below 10% on MMLU-Pro and LongBench, while SoCE reached 38.9% on MMLU-Pro.SoCE was not best on LongBench or TruthfulQA, where it was comparable to other models.Source: §4.2 · p. 7
  • 2.04% relative (0.56% absolute) ∞Bench improvement over the best candidate followed model selection and weight tuning, despite candidates trained on similar data-mix variants.Uniform souping did not regress, and selection alone improved performance 1.49% relatively.Source: §4.2 · p. 7, Table 5 · p. 8
  • SoCE outperformed candidate models and uniform souping on MGSM; weight tuning improved performance 1.57% relative to the best baseline.Uniformly souping all four candidates regressed against MetaMathOctopus-MAPO, while selection reduced that regression.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 Large Scale Model Analysis: BFCL, MGSM, and ∞Bench

  • 52.75% mean accuracy for randomly souped BFCL checkpoints versus 49.85% for unsouped checkpoints; souping also increased category-performance correlations.The unsouped and souped checkpoint standard deviations were 4.7% and 3.9%, respectively.Source: §4.3 · pp. 7–8

4.4 SoCE Candidate selection experiments

  • SoCE gains were substantial on MGSM and BFCL where experts could be identified across anti-correlated checkpoints, but more modest on FLORES-36 or benchmarks with little anti-correlation.The method selects experts for uncorrelated categories and assigns them weights.Source: §4.4 · p. 8

5 Discussion

The discussion frames SoCE as a way to reuse open models and extend capabilities through souping. It points to broader applications while emphasizing low-cost reuse and collaboration opportunities.

5 Discussion

  • SoCE is proposed for adding domains or capabilities to existing open models and addressing overfitting, with gains more modest where clear experts or anti-correlations are lacking.Potential extensions include multilingual and category-level combinations, and use cases where training data remain private but checkpoints can be shared.Source: §5 · p. 8
  • Figure 2 compares category-performance Pearson correlations before and after souping across BFCL, MGSM, and ∞Bench; souped categories become highly linearly correlated.The figure covers approximately 800 souped and unsouped BFCL checkpoints, 40 MGSM checkpoints, and 80 ∞Bench checkpoints.Source: Figure 2 · p. 9
  • The authors present model reuse as a lower-cost opportunity for broader participation and iterative reuse of pretrained models, which can save computing resources.They note that the Llamaderivatives family approaches 150k models with groups of matching architecture and size that can be souped.Source: §5 · p. 9

6 Conclusion

The conclusion presents SoCE as a category-aware souping technique that improves performance and consistency across domains.

6 Conclusion

  • SoCE is a category-aware, principled model-souping technique.The paper frames it as a way to aggregate models.Source: §6 · p. 9
  • The method achieves state-of-the-art results and performance improvement across diverse domains.Source: §6 · p. 9
  • The conclusion reports improved consistency and robustness across benchmark categories.Source: §6 · p. 9

Limitations

SoCE’s evidence is bounded by benchmark structure, evaluation protocol, checkpoint selection, and model lineage. The paper also leaves the scaling behavior of souping insufficiently tested.

General Method Limitations

  • SoCE assumes benchmarks expose explicit subcategories with enough data to estimate per-category scores reliably; discovering implicit structure remains future work.All experiments use predefined category labels.Source: General Method Limitations · p. 10
  • Because none of the evaluated benchmarks provides validation and test splits, candidate selection and weight optimization relied on leaderboard scores, with test-score overfitting remaining a concern.The intended protocol uses validation scores for structure and weight selection, reserving test scores for final reporting.Source: General Method Limitations · p. 10
  • Experiments tested only final post-trained, aligned checkpoints; the authors advise against souping checkpoints from different training stages or unaligned or uncensored models with aligned ones.They cite inheritance of risks as a reason for the latter caution.Source: General Method Limitations · p. 10
  • All experiments use Llama 3 derivatives with essentially the same pretrained checkpoint, leaving it unknown whether souping works across different pretrained checkpoints.Source: General Method Limitations · p. 10

Scaling Laws

  • The paper has not systematically tested souping’s upper bounds or how diminishing returns depend on the diversity and capabilities of the models combined.The authors note that an optimal strategy may vary with capability differences between models.Source: Scaling Laws · p. 10

Ethical Considerations

The appendix reports positive FLORES-36 category-level outcomes for most sampled soups, alongside mixed 70B comparison results. The paper also flags disclosure and safety evaluation as considerations for souped models.

Ethical Considerations

  • The authors recommend disclosing souping procedures because undisclosed procedures can blur mechanistic-interpretability findings, and recommend evaluating souped models for safety and biases.Source: Ethical Considerations · p. 10

A.1 FLORES-36

  • In random pairwise souping of 37 checkpoints fine-tuned on Llama-70B, 35 experiments produced candidates with higher metric scores in more than 20 of 36 FLORES-36 categories and net positive gains across all categories.FLORES-36 is a subset of translations in 18 languages to and from English, used only for ablation studies.Source: §A.1 · p. 21
  • Figure 3 indexes parent-parent-soup triplets on the x-axis; its upper panel counts categories and its lower panel plots BLEU scores, including soup and parent scores.The orange and blue lines count category wins over at least one parent and both parents, respectively.Source: Figure 3 · p. 21

B Comparison of SoCE with TIES and DARE approaches

  • For 70B models, TIES and DARE show similar regressions on IFEval, MMLU-Pro, and LongBench; Truth Q/A gains do not exceed all ingredient models.The final DARE/TIES models outperform two of three ingredient models on Truth Q/A.Source: B Comparison of SoCE with TIES and… · p. 22

C Analysis of TIES and DARE Assumptions

The analysis reports failures in assumptions behind DARE and TIES and relates these findings to performance degradation. It contrasts these methods with SoCE’s more stable results on BFCL-v3.

C Analysis of TIES and DARE Assumptions

  • The authors connect the assumption failures to degradations in Tables 3, 8, and 6, and characterize simple averaging with principled candidate selection as more stable and robust.Source: C Analysis of TIES and DARE Assumptions · p. 22
  • DARE’s near-orthogonal task vectors have a mean pairwise cosine similarity of 0.042, and the authors argue pruning such unaligned vectors destroys shared signal.Source: C Analysis of TIES and DARE Assumptions · p. 22
  • TIES encounters sign conflicts in 52.99% of active parameters, while 81.63% of top-10% magnitude parameters also have conflicts.The paper argues majority voting may not track directional information with fewer candidate models and that high-magnitude parameters are where models disagree most.Source: C Analysis of TIES and DARE Assumptions · p. 22
  • DARE’s assumption that many task-vector parameters can be dropped without meaningful loss is challenged by the risk that indiscriminate dropping disrupts critical or high-risk neurons.Source: C Analysis of TIES and DARE Assumptions · p. 22
  • SoCE consistently outperforms TIES and DARE on BFCL-v3, where degradation is most severe on long-context and deep-reasoning tasks.The authors hypothesize that TIES sign-conflict resolution and DARE rescaling introduce artificial noise affecting weights needed for memory, recall, and extended context.Source: C Analysis of TIES and DARE Assumptions · p. 22, Table 6 · p. 22

D Understanding Impact on Other Benchmarks while souping for one benchmark

The authors evaluate SoCE models tuned for BFCL-v3 on unrelated benchmarks to probe for overfitting. They report that performance is often best or second best, with a maximum reported regression of 1.08% on IFEval for the 8B SoCE model.

D Understanding Impact on Other Benchmarks while souping for one benchmark

E Deep dive on BFCL State-of-the-art Performance: Winrate Analysis of Souped Models and Individual Candidates

SoCE largely retains individual models’ BFCL capabilities and can solve tasks they miss, while threshold sensitivity, Shapley analyses, and examples probe its scope and candidate selection. The analyses also relate expert selection to tractable search and show cases where souping improves outputs.

E Deep dive on BFCL State-of-the-art Performance: Winrate Analysis of Souped Models and Individual Candidates

F Sensitivity Analysis of the Correlation Threshold τ and Computational Upper Bounds of SoCE

G Post-hoc validation of our candidate selection process using Shapley Values

H.1 Wins when candidates fail

  • In BFCL examples, SoCE omits an erroneous month parameter on simple_267 and uses Element.class rather than Element for java_52’s node_type.Source: §H.1 · p. 26

I Souping with Large Language Models with Mixture-of-experts architecture

Baseline pairwise souping experiments applied SoCE principles to Qwen and Llama 4 models on BFCL, with candidate improvements in both model lines. The paper reports that the broader mixture-of-experts study remains future work and lists licenses for the open-source models used.

I Souping with Large Language Models with Mixture-of-experts architecture

J Licenses of Open Sourced Models

Loading 2511.13254v2…