PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie

arXiv:2608.13596v1cs.LGcs.AI

TL;DR

Cross-scale fusion asks whether a small language model can gain from a stronger, architecturally mismatched donor without neuron-wise semantic alignment. APM selects task-relevant donor components using activations, raising the recipient’s average accuracy across 16 benchmarks from 55.5% to 60.6%.

  • Problem

    Heterogeneous fusion often involves additional training, learned transformations, or explicit alignment; whether a task-relevant donor slice can improve a smaller recipient under micro-injection without neuron-level alignment remains an open question.

    Source: Introduction · pp. 1–2

  • Method

    APM uses task-conditioned donor activation maps to select and prune salient components to the recipient architecture, then injects the resulting donor slice with a micro interpolation coefficient.

    Source: Introduction · p. 2

  • Results

    55.5% to 60.6%: across 16 benchmarks, APM raises the original Qwen2.5-3B recipient’s average score.

    Source: Baselines · p. 5

  • Takeaways & Limitations

    Results support a concentration-transfer view in which small-ratio fusion can work without strict neuron-wise semantic correspondence.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

Introduction · Related Work

APM uses task-conditioned donor activations to select and prune recipient-shaped components, then micro-injects them without explicit neuron-wise semantic alignment. The paper reports gains over both the original recipient and Intersection-Merge, including at small injection ratios.

Introduction

  • Classical homogeneous merging uses parameter averaging or task-vector operations, but requires shared architectures and compatible parameterizations; heterogeneous alternatives often add training, learned transformations, or alignment.These constraints motivate transfer across mismatched models without explicit cross-model correspondences.Source: Introduction · pp. 1–2
  • APM profiles task-conditioned donor activations, selects salient layers, dimensions, heads, and MLP neurons, prunes them to recipient shape, and micro-injects the slice into the original recipient.The recipient remains the dominant component, and the pipeline requires neither training nor explicit neuron-wise semantic alignment.Source: Introduction · p. 2
  • APM improves the original 3B recipient and Intersection-Merge across benchmark families, with especially pronounced gains on natural-language-inference and reasoning tasks.Injection-ratio and sequential-fusion analyses indicate activation-guided extraction remains effective with small merge coefficients and can benefit from a subsequent donor injection.Source: Introduction · p. 2

Homogeneous Model Merging

  • Homogeneous merging combines compatible checkpoints or task updates through averaging, Fisher weighting, task-vector arithmetic, trimming, sign resolution, sparsification, or selective updates.Related work also studies task-subspace matching and localization, scaling analyses, and training-free parameter or activation alignment.Source: Homogeneous Model Merging · p. 2

Heterogeneous and Cross-Scale Model Fusion

  • Heterogeneous fusion methods include distillation, learned adapters, correspondence-based alignment, feature-matching fusion, modular skills, and coefficient optimization; APM instead ranks donor-internal components from activations.Unlike many alternatives, APM avoids training, learned intermediates, and cross-model feature or neuron-wise semantic alignment.Source: Heterogeneous and Cross-Scale Model… · p. 2

Intersection-Merge

  • Intersection-Merge truncates leading donor layers and parameter indices to recipient shape, then applies a very small interpolation coefficient without training or explicit semantic alignment.Its static front-aligned extraction is not task-conditioned; APM keeps the injection regime but selects donor components according to the task.Source: Intersection-Merge · pp. 2–3

Activation-Guided Structured Pruning

  • Unlike activation-guided pruning methods that compress standalone models, APM uses task-conditioned donor saliency to select a recipient-shaped slice for micro-injection into an independently trained recipient.Prior structured-pruning approaches score layers, heads, dimensions, or neurons from calibration activations; APM applies activation saliency to transfer-oriented cross-scale fusion.Source: Activation-Guided Structured Pruning · p. 3

Method

APM extracts a task-conditioned donor slice shaped for a smaller recipient, then micro-injects it to transfer task-relevant components without training. The recipient retains its architecture and inference cost.

Preliminaries

  • APM frames transfer as a training-free extraction from a larger donor into a recipient-shaped model, despite differences in depth, width, and attention structure.Source: Problem Formulation · p. 3

Problem Formulation

  • APM replaces Intersection-Merge’s fixed, task-agnostic front-aligned donor selection with task-conditioned extraction while retaining its controlled fusion rule.Intersection-Merge resolves tensor mismatch without training or semantic alignment; APM isolates the effect of selecting a more informative donor slice.Source: Problem Formulation · p. 3

Task-Conditioned Activation Mapping

  • Task-conditioned maps accumulate donor activation magnitudes for layers, hidden channels, attention heads, and MLP neurons over selected task-relevant token positions.Profiling can use prompt, generation, or answer-choice tokens; scores are accumulated online without storing full activation tensors.Source: Task-Conditioned Activation Mapping · p. 3

Activation-Guided Target-Shape Pruning

  • APM prunes to recipient cardinalities, selecting depth by first layers, uniform spacing, or top activation scores, then sorting selected layers to preserve donor-depth order.Source: Activation-Guided Target-Shape Pruning · p. 3
  • APM selects hidden channels globally, MLP neurons per retained layer, and grouped-query heads with separate query/KV indices, then applies indices consistently across coupled tensors.The resulting donor slice is dense and recipient-shaped, preserving tensor compatibility without optimization.Source: Activation-Guided Target-Shape Pruning · p. 3

Micro-Injection Merge

  • A single linear micro-injection keeps the recipient dominant and preserves its inference architecture while letting the selected donor slice bias task-relevant computations.Shape matching eliminates the need for adapters, projections, or further optimization; multiple extracted donors can be composed with small coefficients.Source: Micro-Injection Merge · p. 4

Experiments

APM is evaluated through training-free transfer from larger Qwen2.5 donors into a Qwen2.5-3B recipient across 16 benchmarks. It improves the recipient’s overall average and outperforms or matches the front-aligned IM baseline across the benchmarks.

Benchmark Datasets

  • Evaluation spans 16 benchmarks across six capability families, including mathematics, code generation, instruction following, QA/NLU, commonsense and broad-domain knowledge, and multi-task reasoning.The suites include GSM8K and MATH, HumanEvalPlus and MBPPPlus, IFEval, seven QA/NLU tasks, ARC, HellaSwag, MMLU, and BBH.Source: Benchmark Datasets · pp. 4–5

Baselines

  • APM raises Qwen2.5-3B’s average from 55.5% to 60.6%, exceeding or matching IM (57.8% to 60.6%) on all 16 benchmarks.The gains include RTE (+18.0 pp over 3B and +7.9 pp over IM), QNLI (+13.4 and +10.4 pp), and BoolQ (+8.4 and +6.5 pp).Source: Table 3 · p. 5, Baselines · p. 5

Models

  • Qwen2.5-3B is the recipient, while Qwen2.5-14B and Qwen2.5-32B serve as cross-scale donors from the same model family.Qwen2.5-32B is used for the primary benchmark results, and Qwen2.5-14B for sequential multi-stage fusion analysis.Source: Models · p. 5, Table 2 · p. 5

Experimental Setup

  • Donors are profiled on task-specific activation examples, then reduced to recipient shape using first-n, uniform, or activation-guided layer selection before micro-injection.Profiling can use prompt, generation or choice token positions; profiling examples are disjoint from evaluation examples, and the selected layers retain their original order.Source: Experimental Setup · p. 5

Baselines

  • IM uses front-aligned donor extraction and the same mixing coefficient as APM, controlling for the effect of donor selection rather than the merge rule (Fan et al. 2026).On eight selected BBH subtasks, task-matched APM slices improve over IM on every task, with gains of T2 (+30.0 pp), T1 (+17.6 pp), and T8 (+12.8 pp).Source: Baselines · p. 5

Analysis

The analysis finds that activation-guided donor slices transfer across BBH tasks, generally outperform IM across an effective injection-ratio region, and remain useful in sequential fusion. Donor correctness and activation-set size also affect QNLI results.

Task-Matched Transfer

  • APM slices average 39.9–42.1%, exceeding both baselines; matched activation improves over IM on all eight BBH subtasks by 0.8–30.0 pp.The Qwen2.5-3B student averages 27.3%, and IM at μ = 0.02 averages 32.1%.Source: Task-Matched Transfer · pp. 5–6, Table 4 · p. 6
  • Across BBH-27, the eight APM models average 48.3–49.9%, gaining 4.7–6.3 pp over the student and 3.0–4.6 pp over IM.Each APM variant beats the student on 17–21 tasks and IM on 18–21; off-source slices also improve many other BBH tasks.Source: Table 4 · p. 6, Task-Matched Transfer · p. 6

Injection-Ratio Sensitivity

  • Across BoolQ, MNLI, QNLI, and RTE, APM generally outperforms IM in the effective injection region (0, 0.10), with task-dependent optimal ratios.APM reaches 82.3% on RTE at μ = 0.05, above the 32B donor’s 80.9%, and peaks at 73.9% on QNLI at μ = 0.08.Source: Table 5 · p. 6, Figure 3 · p. 6, Injection-Ratio Sensitivity · p. 7

Task-Matched Transfer

  • Both raw-score and normalized-gain tests show a significant matched-task advantage, while several off-diagonal entries are column maxima.The results indicate that matched activation usually yields the strongest donor slice, alongside effective cross-task transfer.Source: Task-Matched Transfer · pp. 6–7

Sequential Multi-Stage Fusion

  • With two donor injections at μ = 0.02, APM raises the seven-task average from 64.3% to 68.2% and then 69.9%, while IM reaches 66.4% and 66.7%.The second-stage gain is 1.7 pp for APM versus 0.3 pp for IM; APM-S2 leads on MNLI, RTE, QNLI, BoolQ, and COPA, while APM-S1 leads on PIQA.Source: Table 6 · p. 7, Sequential Multi-Stage Fusion · p. 7

Sensitivity to Activation-Data Selection

  • On QNLI at μ = 0.05, donor-correct subsets average 58.5%, versus 56.9% for random selection across nine variants.The 25-example set scores 59.6%, compared with 58.6% for 200 examples and 57.3% for 1,000; recipient-solved examples remain useful when the donor is also correct.Source: Table 7 · p. 7, Sensitivity to Activation-Data Selection · p. 7

Conclusion

APM uses task-conditioned activation profiling and structured component selection to improve cross-scale fusion while retaining the recipient architecture and micro-injection rule. The conclusion presents this lightweight, training-free approach as evidence that heterogeneous models can transfer capabilities without explicit neuron-wise semantic alignment.

Conclusion

  • APM replaces fixed donor truncation with task-conditioned activation profiling and structured component selection, improving recipients over IM under the same interpolation rule.Fine-grained BBH transfer, gains across small injection ratios, and continued gains under sequential fusion demonstrate the quality and composability of the extracted donor slices.Source: Conclusion · p. 7
  • Lightweight activation statistics and target-shape selection yield substantial gains while preserving recipient architecture, inference cost, and the training-free workflow.The results suggest potential for fusing heterogeneous models without explicit neuron-wise semantic alignment.Source: Conclusion · p. 7
  • Separating target-shape reduction from activation-guided selection keeps the recipient architecture and micro-injection rule unchanged, facilitating adaptation across benchmark families.The paper positions APM as a lightweight complement to model scaling.Source: Conclusion · p. 7

Supplementary Material · Complete BBH-27 Transfer Results

The supplement presents complete BBH-27 per-task accuracy results for the original recipient, the Intersection-Merge baseline, and eight APM slices, with all evaluations using an injection ratio of µ = 0.02.

Complete BBH-27 Transfer Results

  • Table 3 covers per-task accuracy across all 27 BBH tasks.The table caption reports 6,511 evaluation examples in total.Source: Table 3 · p. 10
  • The 3B column denotes the original Qwen2.5-3B recipient, while IM denotes the Intersection-Merge baseline.Scores are reported as percentages.Source: Table 3 · p. 10
  • Columns T1–T8 report the eight APM slices defined in main-paper Table 4(a).Source: Table 3 · p. 10
  • The final row gives the unweighted macro-average across the 27 tasks.The caption says these averages reproduce those summarized in Table 4(c).Source: Table 3 · p. 10
  • All reported results use an injection ratio of µ = 0.02.Source: Table 3 · p. 10
  • The caption identifies N as the number of evaluation examples in each subtask.Source: Table 3 · p. 10

MATH-500 Significance Test · Complete Injection-Ratio Results

The supplied passages show column labels for a MATH-500 significance test and complete injection-ratio results, but provide no result values.

MATH-500 Significance Test

Complete Injection-Ratio Results

Detailed Activation-Data Selection Results

The supplied passages describe two QNLI summaries across nine variants per condition: maximum accuracy, and mean accuracy with normalized variance.

Detailed Activation-Data Selection Results

  • The figure reports maximum QNLI accuracy across nine variants per condition and mean accuracy with normalized variance across those variants.The supplied captions provide no underlying values or comparisons.Source: Detailed Activation-Data Selection… · p. 11

Detailed APM Formulation

APM profiles donor activations to select layers, hidden channels, attention heads, and MLP neurons sized for the recipient. It then uses structured extraction and micro-injection to transfer the selected donor components.

Detailed APM Formulation

  • APM selects donor layers, hidden channels, attention heads, and MLP neurons according to activation scores and the recipient’s required dimensions.The passage identifies layer outputs, hidden-channel activations, attention-head outputs, and postactivation MLP neuron values as the profiled quantities.Source: Detailed APM Formulation · p. 12
  • The selected MLP neurons are the top ms entries of the MLP score set.Source: Detailed APM Formulation · p. 12
  • APM denotes the collection of selected components as S∗ and uses it for structured extraction.Source: Detailed APM Formulation · p. 12
  • For K donor slices, multi-donor micro-injection uses coefficients {µq} whose total is less than 1.Source: Detailed APM Formulation · p. 12

Statistical Analysis of Diagonal Effects

The analysis tests whether activation sources matched to evaluation tasks yield higher performance than unmatched sources. It uses exact permutation tests and complementary IM-relative and column-standardized analyses.

Statistical Analysis of Diagonal Effects

  • Matched-task performance is the diagonal mean of a cross-task accuracy matrix, while unmatched-task performance is the mean of its off-diagonal entries.Rows index activation sources and columns index evaluation tasks in the same order.Source: Statistical Analysis of Diagonal Effects · p. 12
  • The one-sided exact permutation test evaluates positive diagonal advantage under exchangeability, using 40,320 permutations for BBH-8 and 5,040 for QA-7.The analysis exhaustively enumerates permutations rather than using Monte Carlo sampling.Source: Statistical Analysis of Diagonal Effects · p. 12
  • APM results are expressed as gains over IM; subtracting that common baseline within each task leaves the unstandardized test statistic unchanged.Column-wise z-scores additionally give equal scale to tasks with different across-source variation, testing matched activation relative to each task’s typical variation.Source: Statistical Analysis of Diagonal Effects · p. 12

Related papers

Loading 2608.13596v1…