Source-linked AI summary
FusionEval: Benchmarking Model Fusion for Large Language Models
Shuo Cai, Yanggan Gu, Yuhang Liu, Su Lu, Yuanyi Wang, Yang Yu, Zhen Li, Zhitian Hou, Guanghao Zhu, Ming Li, Hongxia Yang
TL;DR
Existing evaluations provide limited coverage of long-chain reasoning and tool use, despite the need to assess whether these capabilities can be retained together. FusionEval compares fusion methods on controlled expert pools, and its experiments find that tool use is vulnerable under training-free fusion while MOPD produces a more balanced profile at additional cost.
Problem
Existing fusion evaluations provide limited coverage of long-chain reasoning and tool use, leaving their joint retention and computational cost insufficiently examined.
Source: §1 · pp. 1–2Method
Results
Training-free methods retain reasoning performance but substantially degrade tool use, while MOPD retains tool use and achieves a more balanced profile at both tested Qwen3 scales.
Source: §1 · pp. 2–3, §5 · p. 10Takeaways & Limitations
Jointly retaining reasoning, instruction following, and tool use at low computational cost remains a challenge.
Source: §5 · p. 10MOPD requires additional data preparation, inference, and student training compared with training-free fusion.
Source: §4.4 · p. 9
Abstract
from arXiv · showhide
Model fusion aims to integrate the capabilities of multi-expert models into a single model. Existing evaluations offer limited coverage of reasoning with long chains of thought and tool use, while strong average performance can mask substantial losses in individual expert capabilities. We introduce FusionEval, a benchmark that provides controlled expert pools, integrates fusion methods with and without additional training, and offers a unified evaluation suite. The pools contain domain experts independently trained from Qwen3 and Llama base models across math, code, science, instruction following, and tool use. The suite evaluates capability retention and computational cost on eight public benchmarks under comparable data access rules, reporting normalized average and worst performance, runtime, and peak GPU memory usage. Extensive experiments show that training-free methods preserve reasoning performance but substantially degrade tool use at both model scales. In contrast, MOPD preserves tool use and achieves a more balanced capability profile, but requires additional data preparation, inference, and training. Jointly retaining reasoning, instruction following, and tool use at low computational cost remains a challenge. We hope FusionEval supports future research on efficient model fusion that jointly preserves expert capabilities. Our code is available at https://github.com/Baicaihaochi/FusionEval_Benchmark.
1 INTRODUCTION
Existing fusion evaluations provide limited coverage of long-chain reasoning and tool use, motivating a benchmark that tests their joint retention and computational cost. FusionEval supplies controlled expert pools and a unified evaluation suite; experiments find uneven capability retention and a more balanced profile from MOPD at additional cost.
Motivation
- Existing evaluations offer limited coverage of experts trained on long chain-of-thought responses and tool use, leaving their joint retention insufficiently tested.A complementary setting is needed to examine simultaneous capability retention and computational cost.Source: §1 · pp. 1–2
Benchmark
- FusionEval compares methods with and without additional training using independently fine-tuned experts across five domains and eight public benchmarks under consistent data-access rules.It reports normalized mean and minimum benchmark performance, runtime, and peak GPU memory.Source: §1 · p. 2, Figure 1 · p. 2
Key findings
- High average scores can mask severe tool-use losses, so strong reasoning retention does not establish joint retention across reasoning, instruction following, and tool use.The authors identify efficient joint retention as a continuing research challenge.Source: §1 · pp. 2–3
- MOPD achieves the highest minimum normalized benchmark score at both tested Qwen3 scales, while training-free methods retain math and code better than tool use.MOPD requires additional data preparation, inference, and training.Source: §1 · pp. 2–3
2 RELATED WORK
Prior work spans multiple fusion techniques and evaluation designs, but coverage of long-chain-of-thought experts and tool use remains limited. FusionEval addresses this gap with independently trained domain experts and consistent comparisons of retention and computational cost.
Model Fusion Methods
- Fusion methods range from training-free parameter operations and data-assisted combination to training-based calibration and multi-teacher distillation.The methods differ in how they combine updates, calibrate representations, or optimize a student.Source: §2 · p. 3
Model Fusion Evaluations
- Existing evaluations vary in method support, expert selection, and task coverage, with prior task coverage expanding from conventional NLP to math and code.Evaluations include both community-model selection and controlled domain experts trained under a common protocol.Source: §2 · p. 3
- Long-CoT expert retention and tool use remain insufficiently studied, motivating comparisons of independently trained experts under consistent data-access rules and tasks.FusionEval reports mean and worst-benchmark retention alongside computational cost.Source: §2 · p. 3
3 FUSIONEVAL
FusionEval evaluates expert-capability retention and computational cost across five task categories using controlled expert pools and a common protocol. It compares fusion methods on eight benchmarks and reports normalized average and worst-benchmark performance, runtime, and peak GPU memory.
3 FUSIONEVAL
- FusionEval uses independently trained domain experts and a common evaluation protocol to measure capability retention and fusion computation.The evaluation covers eight public benchmarks and reports mean and worst-benchmark normalized performance, runtime, and peak GPU memory.Source: §3 · p. 3
3.1 TASK SCOPE
- The five task categories are math, code, science question answering, instruction following, and tool use, testing reasoning and constraint-following capabilities.Tool use evaluates correct calls to external tools; science, math, and code assess long-CoT reasoning.Source: §3.1 · pp. 3–4
3.2 EXPERT POOL CONSTRUCTION
- Each pool contains five domain experts independently SFT-trained from the same base, with shared architecture and tokenizer; training data is cleaned and evaluation duplicates are removed.Table 2 lists the domain SFT datasets and post-preprocessing sample counts.Source: §3.2 · p. 4, Table 2 · p. 4
3.3 EVALUATION PROTOCOL
- The evaluation compares weight-combination, data-assisted, and distillation methods under shared prompts, decoding, and scoring rules, including multi-turn tool interactions.Tables 3 and 4 summarize method operations and data requirements, and benchmarks by domain, respectively.Source: §3.3 · p. 4, Table 3 · p. 5, Table 4 · p. 5
3.4 METRICS
- Avg. Norm. is mean performance relative to domain experts, Min. Norm. is the worst normalized benchmark, and fusion cost is measured by runtime and peak GPU memory.Benchmark scoring includes accuracy, pass@1, strict accuracy, and microaveraged success; a normalized score of 1 matches the corresponding expert.Source: §3.4 · pp. 4–5
4 EXPERIMENTS
Across the experiments, MOPD achieves the highest minimum normalized score at both tested Qwen3 scales, while training-free methods often lose tool-use performance. Continual fusion, computational cost, capability tradeoffs, and calibration analyses show that balanced retention remains challenging.
4.1 MULTI-DOMAIN FUSION PERFORMANCE
- MOPD has the highest Min. Norm. at 0.708 on 4B and 0.859 on 8B, while Seq-KD trails MOPD on all eight benchmarks in both pools.MOPD retains Student’s strong tool-use performance while scoring higher in reasoning and instruction following; its Avg. Norm. is 0.899 and 0.955, respectively.Source: Table 5 · p. 6, Table 6 · p. 6, §4.1 · p. 7
- BFCLv3 is the weakest normalized benchmark for seven of eight training-free methods on 4B and all eight on 8B; larger scales do not ensure better retention.On 8B, even L&S reaches only 39.1 BFCLv3 versus 67.5 for the corresponding expert; TA’s Avg. Norm. rises from 0.908 to 0.961 while its Min. Norm. falls from 0.472 to 0.466.Source: §4.1 · pp. 6–7
4.2 CONTINUAL FUSION
- Continual Soup and DELLA finish 12.1% and 11.8% below their joint-fusion references in normalized average performance.Adding the tool-use expert yields little tool-use improvement for both methods while degrading math, code, and instruction following; DELLA’s minimum-score decline is 2.3%, versus 31.2% for Soup.Source: §4.2 · p. 7, Table 7 · p. 7
4.3 ANALYSIS
- On Qwen3-4B, TA takes 0.62 minutes and has higher Avg. Norm. than slower DELLA and L&S, while DELLA uses 26.48 GiB peak memory versus TIES’s 16.70 GiB.MOPD training takes about 10.18 and 12.82 hours on 4B and 8B, compared with Seq-KD’s 16.11 and 23.99 hours, including offline teacher-response generation and student training.Source: §4.3 · p. 8, Figure 2 · p. 8
- BFCLv3 performance does not track a single update statistic consistently, and gains in tool use can coincide with losses on reasoning benchmarks.F2–F4 outperform F1 on BFCLv3 but score lower on AIME25 and LCBv5; from F3 to F4, BFCLv3 and AIME25 improve while LCBv5 declines further.Source: Figure 3 · p. 8, §4.3 · p. 8, Figure 4 · p. 9
- Neither FeatCal nor Surgery improves both average and worst-benchmark performance under the tested configurations.On 8B, Surgery raises Min. Norm. by 0.042 but lowers Avg. Norm. by 0.068; FeatCal changes neither metric appreciably.Source: Table 8 · p. 9, §4.3 · p. 9
4.4 FUTURE DIRECTIONS
- Future work targets joint capability retention in training-free fusion, lower-cost MOPD, and continual fusion that acquires new capabilities without losing existing ones.Suggested directions include selecting tool-use updates by separate validation data, selectively distilling weaker domains, and testing layer- or parameter-level update preservation.Source: §4.4 · p. 9
5 CONCLUSION
FusionEval evaluates fusion across five expert domains and reports capability retention and computational cost. Its experiments expose tool-use losses and the costs of MOPD, while continual fusion underperforms joint fusion in the reported experiments.
5 CONCLUSION
- FusionEval compares training-free and training-based methods across eight public benchmarks, measuring capability retention, runtime, and peak GPU memory.Its expert pools cover math, code, science reasoning, instruction following, and tool use, using Qwen3 and Llama models.Source: §5 · p. 10
- Several methods retain reasoning performance but lose tool-use capability; MOPD retains tool use and has a more balanced profile, at the cost of data preparation, inference, and training.The reported tool-use vulnerability means high average scores alone do not establish joint capability retention.Source: §5 · p. 10
- Continual Fusion strengthens some existing capabilities as experts arrive but loses overall model capabilities relative to joint fusion in these experiments.Source: §5 · p. 10
- The authors will release the data, expert pools, fusion scripts, and configurations to support comparisons under the same protocol.Source: §5 · p. 10
A EXPERT POOL RESULTS
The section provides cross-domain expert-score tables for Qwen3-4B and Qwen3-8B; the captions identify column maxima but do not state their values.
A EXPERT POOL RESULTS
- Table 9 reports cross-domain expert scores for Qwen3-4B, with boldface marking the maximum in each column.Source: Table 9 · p. 13
- Together, the tables cover expert scores at two Qwen3 model sizes, but their captions alone do not establish which domain scores highest.Source: Table 9 · p. 13, Table 10 · p. 13
- Table 10 reports cross-domain expert scores for Qwen3-8B, with boldface marking the maximum in each column.Source: Table 10 · p. 13
A.1 LLAMA-3.2-3B EVALUATION
The Llama-3.2-3B evaluation compares six parameter-fusion methods on the expert pool. All fall below their corresponding experts across the eight benchmarks, with tool use especially weak for most methods.
A.1 LLAMA-3.2-3B EVALUATION
- The Llama-3.2-3B comparison evaluates TA, TIES, DARE, DELLA, Consensus TA, and TA+FeatCal; method configurations are listed in Appendix Table 21.Source: §A.1 · p. 13
- Tool use is the least-retained capability for most methods, while TIES scores highest on tool use but lowest on both code benchmarks.BFCLv3 is the lowest normalized benchmark for five of the six methods.Source: §A.1 · p. 13
- All six evaluated fusion methods fall below their corresponding domain experts on all eight Llama benchmarks, including substantial losses in math, code, and science reasoning.DELLA exceeds experts on all five reasoning benchmarks on Qwen3-8B but falls below them on all five on Llama; on Llama it has the highest Avg. Norm. (0.568) and a Min. Norm. of 0.114.Source: §A.1 · p. 13, Table 12 · p. 14
B.1 EXPERT SFT SETTINGS
- The appendix identifies a table of target training settings for domain-expert supervised fine-tuning.Source: Table 13 · p. 14
B.2 NOTATION
- The appendix includes a notation table listing symbols and their definitions.Source: Table 14 · p. 14
B.3 DATA DECONTAMINATION
The decontamination procedure uses exact matching to remove training samples, but this filter does not catch paraphrases or pretraining contamination.
B.3 DATA DECONTAMINATION
- Exact-match filtering does not detect paraphrases.Source: §B.3 · p. 14
- Exact-match filtering also does not detect contamination in pretraining data.Source: §B.3 · p. 14
- Exact matching is used to remove training samples, and Table 15 reports the counts removed for each benchmark.Source: Table 15 · p. 15
B.4 EVALUATION PROTOCOLS
The appendix specifies evaluation and efficiency protocols, tuning and selected configurations, update-geometry analyses, and additional fusion results. It documents measurement boundaries and the scope of the reported comparisons.
B.4 EVALUATION PROTOCOLS
- Quick evaluation uses one generation per question and three BFCLv3 questions from each of 17 subsets; full evaluation uses the same fixed snapshots, so it is not an independent held-out test.Quick and full evaluations use seed 42 and top-k of 20; Tables 16 and 17 summarize their sampling and decoding settings.Source: §B.4 · p. 15, Table 16 · p. 15, Table 17 · p. 15
B.5 EFFICIENCY MEASUREMENT DETAILS
- Distillation runtime sums recorded rollout and training stages for MOPD and teacher generation and student training for Seq-KD, under a planned budget of 50 updates per method.Measurements use H800 GPUs; parameter-fusion timings have stated exclusions, and some timings use configurations other than the selected evaluation checkpoints.Source: §B.5 · p. 15
B.6 HYPERPARAMETER TUNING
- Search cost is estimated as mean algorithm runtime multiplied by validation count, rather than by timing a complete search.Candidate counts exclude full evaluations and internal optimization steps; zero denotes a fixed configuration, and Table 18 lists search spaces and candidate counts.Source: §B.6 · pp. 15–16, Table 18 · p. 16
B.7 SELECTED FUSION CONFIGURATIONS
- Selected configurations are reported for the Qwen3-4B, Qwen3-8B, and Llama-3.2-3B expert pools; calibration uses 128 examples per domain and distillation uses a constant learning rate.Tables 19–21 list the configurations corresponding to the reported models in each pool.Source: §B.7 · p. 16, Table 19 · p. 17, Table 20 · p. 17
B.8 UPDATE GEOMETRY DETAILS
- Update geometry compares Qwen3-4B backbone updates relative to the shared base, excluding embeddings and output heads, using update L2 norm, Agent cosine similarity, and Agent-weighted sign conflict.The zero-update fraction uses the same denominator as sign conflict, with the condition that the fused update is zero.Source: §B.8 · p. 18
- Fisher candidates with full BFCLv3 scores of 57.9, 45.1, and 29.9 are compared on two questions, while candidate settings and quick-evaluation samples are specified separately.Table 23 summarizes responses and outcomes; Table 22 reports sign-conflict and zero-update fractions.Source: §B.8 · p. 18, Table 23 · p. 18, Table 22 · p. 18
B.9 ADDITIONAL FUSION RESULTS
- Additional fusion and calibration results on Qwen3 are reported with scores following Tables 5 and 6.Source: Table 24 · p. 19
B.10 CONTINUAL FUSION RESULTS
This section gives the update rules for Continual Soup and Task Arithmetic, then reports per-benchmark scores at each fusion stage.
B.10 CONTINUAL FUSION RESULTS
- Continual Soup assigns equal weight to the current model and incoming expert at each step.Source: §B.10 · p. 19
- Task Arithmetic starts from the instruction-following expert and adds incoming experts’ task vectors relative to the fixed base, with λ = 0.3 for stages two onward.The stage sequence is IF, then Math, Science, Code, and Agent.Source: §B.10 · p. 19
- Per-benchmark scores (%) are reported for each Continual Fusion stage on Qwen3-4B.Source: Table 25 · p. 19
C DISTILLATION DATA
The appendix describes the distillation prompt sources and preprocessing counts, along with how prompts are selected and used by MOPD and Seq-KD.
C DISTILLATION DATA
- MOPD uses prompts with an 8,192-token limit, retaining tool definitions and dialogue history for Agent prompts without executing tool calls during training.Seq-KD uses the fixed 25,600 prompts scheduled for its first 50 steps; both model sizes use the same prompts, with responses from corresponding 4B or 8B experts.Source: C DISTILLATION DATA · p. 20, Table 26 · p. 20