PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer

Jiahe Fan, Si Chen, Yinghao Hou, Wenbo Xia, Ke Xu, Hong Xie, Enhong Chen

arXiv:2609.39369v1cs.CLcs.LG

TL;DR

Specialist behavior is usually transferred to general language models through training, distillation, or alignment, motivating a direct parameter-level alternative. This paper applies training-free IM and APM to project specialist donors into a general recipient and inject their backbones without semantic alignment. Both methods improve the recipient across specialist roles, with their reported strengths differing by task.

  • Problem

    Transferring task-specific behavior to a general language model usually requires training, distillation, or representation alignment, motivating a direct parameter-level alternative.

    Source: §1 · p. 1

  • Method

    The paper applies training-free IM and APM to project specialist donors into a recipient-shaped backbone and inject parameters without semantic alignment.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    APM improves the recipient across BEIR, reward, and code tasks; IM improves retrieval and reward, matches APM on reward Overall, and is complementary on reranking.

    Source: §7 · p. 4

  • Takeaways & Limitations

    The reported results support training-free transfer across specialist roles, including embedding, reranking, reward modeling, and code.

    Source: §1 · p. 1, §7 · p. 4

Abstract

from arXiv · show

Specialized models encode task-oriented behavior, but transferring that behavior to a general language model usually requires training, distillation, or representation alignment. We study whether such ability can instead be transferred directly at the parameter level. We apply two existing training-free heterogeneous merging methods, previously shown to transfer knowledge between general language models, to specialist-to-general transfer, projecting a specialist donor into the recipient's shape and interpolating backbone parameters without gradient updates or semantic alignment. Intersection-Merge (IM) injects a prefix-aligned donor slice matching the recipient shape, while Activate-Prune-Merge (APM) uses forward-pass activation statistics to select which donor dimensions to retain before injection. Across embedding, reranking, reward modeling, and MoE code-specialist transfer, both methods improve the general recipient, showing that simple heterogeneous merging can move capabilities across diverse specialist roles.

1. INTRODUCTION

The paper asks whether specialist behavior can be transferred directly into a general model through heterogeneous parameter injection, without training or semantic alignment. It tests two training-free methods across embedding, reranking, reward modeling, and code-specialist transfer.

1. INTRODUCTION

  • Specialists serve distinct task roles, while transferring their behavior usually requires supervised fine-tuning or distillation; parameter-space merging offers a different route.The cited transfer routes require optimization and often examples connecting teacher outputs to student inputs, whereas merging can manipulate parameters without additional inference cost.Source: §1 · p. 1
  • The study tests IM and APM with Qwen3 embedding, reranking, reward, and MoE-coder donors projected into Qwen2.5-3B, without training or semantic alignment.Evaluations cover BEIR, RewardBench, and code benchmarks.Source: §1 · p. 1
  • Noise controls support donor-structure effects: prefix cropping misses sparsely routed code experts, while activation-guided selection retains them.The authors frame specialist-to-general transfer as heterogeneous merging across mismatched functions, not only model scale.Source: §1 · p. 1

2. RELATED WORK

The related work distinguishes heterogeneous merging from approaches that rely on optimization, shared architectures, or alignment. It also motivates applying these methods to specialists with different task roles and structures.

2.1. Model Merging and Heterogeneous Transfer

  • Prior merging methods often assume related architectures, while training-free alignment approaches retain architecture or feature-alignment constraints; IM and APM were previously used for general causal models differing in scale [20] [16] [17].Figure 1 depicts their application to embedding, reranking, reward, and MoE/Coder donors projected into one Qwen2.5-3B recipient.Source: §2.1 · pp. 1–2, Figure 1 · p. 2

2.2. Pruning and Knowledge Transfer

  • APM draws on activation-aware pruning to select donor structures before injection, while keeping the recipient frozen and avoiding representation matching [17].The selection uses activation statistics rather than coordinate positions alone.Source: §2.2 · p. 2

2.3. Specialist Model Roles

  • Embedding, reranking, reward, and code specialists target different tasks; the study tests whether their task bias survives projection, including sparse-MoE-to-dense transfer.Embeddings support retrieval, rankers score query–document relevance, reward models score preferences, and code specialists target program synthesis.Source: §2.3 · p. 2

3. METHOD

The method projects specialist backbones into the recipient’s shape and interpolates selected tensors while retaining the recipient’s embeddings and language-model head. IM selects coordinates by prefix intersection; APM uses donor activations to choose structures before injection.

3. METHOD

  • IM and APM inject projected backbone tensors into the recipient through a projection rule and injection ratio, while token embeddings and the language-model head remain from the recipient.The evaluation spans reranking, embedding, reward preference scoring, and code generation using the corresponding specialist donors.Source: §3 · p. 2, §3.3 · p. 3

3.1. Intersection-Merge

  • IM copies each donor tensor’s leading prefix intersection into a recipient-shaped shell, discarding layers and coordinates outside recipient depth and width.This deterministic coordinate choice requires neither calibration data nor gradients, and can omit specialist coordinates outside the prefix.Source: §3.1 · p. 2

3.2. Activation-Guided Pruning

  • APM ranks donor coordinates by mean absolute activation, selecting hidden channels once and MLP neurons separately in each retained layer.It also selects query and key–value heads using activation scores, preserves donor order, and extracts recipient-shaped slices; activations do not update parameters.Source: §3.2 · p. 3

3.3. Specialist-to-General Transfer

  • The transfer framework evaluates Qwen3 reranker and embedding donors on retrieval, a Skywork reward donor on pairwise preferences, and a sparse MoE Qwen3-Coder donor on code generation.Each specialist is evaluated on its corresponding task.Source: §3.3 · p. 3

4. EXPERIMENTAL SETUP

Experiments use Qwen2.5-3B as the recipient and task-specific Qwen3, Skywork, and MoE/Coder donors. They evaluate retrieval, reward modeling, and code generation under defined scoring protocols and a shared injection-ratio sweep.

4.1. Models

  • The 3.09B-parameter, 36-layer Qwen2.5-3B recipient is paired with Qwen3 reranker and embedding, Skywork reward, and sparse MoE Qwen3-Coder donors.The donor models range from 8B to 30B-A3B as named in the model descriptions.Source: §4.1 · p. 3

4.2. Benchmarks

  • Retrieval is evaluated on 11 BEIR datasets, including ArguAna, FiQA, HotpotQA, NQ, and TREC-COVID.The full dataset list also includes CQADupStack, DBPedia-Entity, NFCorpus, Quora, SciFact, and Touche-2020.Source: §4.2 · p. 3

4.3. Evaluation Protocol

  • Retrieval uses NDCG@10; reranking scores TF–IDF candidates by yes/no log probabilities, embedding uses weighted-mean pooling with L2 normalization, and reward and code use their stated benchmark measures.APM calibration sequences locate coordinates only and are disjoint from test sets.Source: §4.3 · p. 3

4.4. IM/APM Search Spaces

  • IM, APM, and Noise are evaluated over α ∈{0.005, 0.010, 0.015, 0.020, 0.025}, with main tables reporting one locked checkpoint and additional envelope results.Table 1 covers reranker NDCG@10 (%) in locked and per-dataset oracle-envelope forms.Source: §4.4 · p. 3, Table 1 · p. 3

5. RESULTS

Across reranking, embedding, reward modeling, and code generation, specialist-to-general parameter injection improves task performance. Gaussian-noise controls generally remain below the strongest merging results.

5.1. Reranker Results

  • Average reranking NDCG@10 rises from 13.75% to 14.92% (IM) and 15.17% (APM), with envelope means of 18.00% and 18.82%.TREC-COVID reaches 50.86 under APM in the envelope.Source: §5.1 · p. 3

5.2. Embedding Results

  • Average embedding NDCG@10 rises from 11.61% to 12.06% (IM) and 12.21% (APM), led by SciFact at 26.64% (APM).TREC-COVID rises from 12.63% to 14.20% (IM).Source: §5.2 · p. 4, Table 2 · p. 4

5.3. Reward and MoE/Coder Results

  • RewardBench Overall rises from 76.83% to 77.41%, while MoE/Coder APM raises the mean from 58.61% to 60.18%.RewardBench Chat reaches 68.71% (IM), and code results include 38.76% on CodeComplex and 81.71% on HumanEval (APM).Source: §5.3 · p. 4, Table 3 · p. 4, Table 4 · p. 4

5.4. Ablation Against Gaussian Noise

  • Gaussian noise remains below locked APM on average retrieval, RewardBench Overall, and the reported HumanEval, MBPP, and CodeComplex controls.Noise can still help on individual datasets.Source: §5.4 · p. 4

6. ANALYSIS

Related Qwen backbones appear to retain compatible low-level structure, allowing small α to inject specialist bias without overwriting the recipient.

6. ANALYSIS

  • APM’s activation-guided selection helps reranking, embedding, and code, while IM remains competitive for reward and some reranking settings.The preferred α varies across tasks and datasets; on MoE code, prefix cropping misses routed experts, leaving locked IM below the recipient (57.96 vs. 58.61).Source: §6 · p. 4

7. CONCLUSION

IM and APM transfer specialist behavior without training or semantic alignment, improving the general recipient across several specialist roles.

7. CONCLUSION

  • Across BEIR, reward, and code tasks, APM improves the recipient; IM improves retrieval and reward and is complementary on reranking.The authors conclude that these results support training-free transfer across specialist roles.Source: §7 · p. 4

A. ORACLE ENVELOPES

The oracle envelopes report best-grid results for reranking, embedding, and MoE/code benchmarks. These per-dataset or per-benchmark scores are not a single deployable checkpoint.

A. ORACLE ENVELOPES

  • Each IM, APM, and Noise score is the best value on the same five-ratio α grid, and the tables do not represent one deployable checkpoint.APM also varies activation position.Source: A. ORACLE ENVELOPES · p. 5
  • Reranker NDCG@10 (%) is reported per dataset using the best α from a five-point grid.The table identifies Qwen3-Reranker-8B as the donor.Source: Table 5 · p. 5, A. ORACLE ENVELOPES · p. 5
  • Embedding NDCG@10 (%) is reported per dataset using the best α from a five-point grid.The table identifies Qwen3-Embedding-8B as the donor.Source: Table 6 · p. 5, A. ORACLE ENVELOPES · p. 5
  • MoE/Coder results (%) are reported per benchmark using the best α from an oracle envelope.The table identifies Qwen3-Coder-30B-A3B-Instruct as the donor.Source: Table 7 · p. 5

Related papers

Loading 2609.39369v1…