PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie

arXiv:2608.13596v1cs.LGcs.AI

TL;DR

The paper asks whether a stronger donor can improve a smaller recipient without neuron-level semantic alignment. It proposes training-free Activation-Prune-Merge, which selects task-relevant donor components before micro-injection and raises average accuracy from 55.5% to 60.6% across 16 benchmarks.

  • Problem

    The paper asks whether stronger donors can improve smaller recipients without neuron-level semantic alignment, amid heterogeneous-transfer methods requiring training or learned transformations.

  • Method

    APM uses task-conditioned donor activation profiling to select and prune salient components into the recipient architecture before micro-injection.

  • Results

    55.5% to 60.6%: APM raises the original 3B recipient’s average accuracy across 16 benchmarks and improves over IM across benchmark families.

  • Takeaways & Limitations

    The results support concentration transfer, in which small-ratio fusion relies on concentrated task-relevant donor information rather than strict neuron-wise correspondence.

Abstract

from arXiv · show

Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

Introduction · Related Work

The paper positions APM as a training-free solution to cross-scale heterogeneous fusion, replacing architecture-constrained or alignment-heavy approaches with task-conditioned activation-guided pruning and micro-injection. It builds on Intersection-Merge’s evidence that a target-shaped donor slice can transfer capability without explicit semantic alignment.

  • Introduction: Classical homogeneous merging requires shared architectures and compatible parameterizations, limiting which models can be merged directly.These methods include parameter averaging, vector arithmetic, conflict resolution, and sparsification.
  • Heterogeneous and Cross-Scale Model Fusion: Heterogeneous transfer methods use distillation, adapters, projections, or alignment, but often require training, learned transformations, optimization, or explicit correspondences.These approaches address mismatched architectures or scales without directly combining incompatible parameters.
  • Introduction: APM profiles task-conditioned donor activations, selects salient layers, hidden dimensions, attention heads, and MLP neurons, then prunes them into the recipient’s exact architecture.The framework is training-free and avoids explicit neuron-wise semantic alignment.
  • Introduction: APM improves cross-scale transfer beyond IM under the same micro-injection rule, indicating that activation-guided pruning extracts a more effective, task-relevant donor slice.The paper frames this as a concentration-transfer view in which small-ratio fusion adds a concentrated donor extract rather than performing neuron-wise transplantation.
  • Homogeneous Model Merging: Homogeneous model merging combines parameters or task updates across shared architectures, including checkpoint averaging, Fisher-weighted merging, and task-vector methods.The related work is supported by low-loss paths and linear mode connectivity between compatible solutions.
  • Intersection-Merge: Intersection-Merge truncates a large donor to the recipient architecture and applies a tiny interpolation coefficient, transferring capability without training or explicit semantic alignment.Its static front-aligned extraction is not conditioned on the evaluated task.
  • Activation-Guided Structured Pruning: Activation-based pruning estimates structured importance from calibration activations, whereas APM profiles a task-conditioned donor, extracts a recipient-shaped slice, and micro-injects it.Minitron, SliceGPT, and Wanda instead target standalone compressed-model pruning or complementary activation-aware pruning strategies.

Method

APM is a training-free cross-scale fusion method that profiles task-conditioned donor activations, prunes salient components to the recipient’s architecture, and merges the resulting slice through micro-injection. It preserves the recipient’s architecture and inference cost without semantic alignment or further optimization.

  • Extraction: APM replaces fixed, task-agnostic front-aligned donor selection with a task-conditioned extraction operator that produces a recipient-shaped donor slice.This isolates the effect of selecting a more informative donor slice while retaining the controlled fusion rule.
  • Activation profiling: Activation profiling accumulates donor activation magnitudes over layers, hidden channels, attention heads, and MLP neurons at task-relevant token positions.Profiling may use prompt, generation, or answer-choice tokens depending on the task, and scores are accumulated online without storing full activation tensors.
  • Target-shape pruning: APM selects recipient-sized depth, hidden, attention, and MLP dimensions using first-n, uniform, or activation-guided depth strategies and activation scores for the other dimensions.Selected layer indices remain in donor-depth order, while grouped-query attention treats query and key–value heads separately.
  • Target-shape pruning: Coupled tensor index sets are applied consistently across embeddings, normalization parameters, projections, and MLP tensors, yielding a dense compatible donor slice without optimization.This preserves grouped-query structure and matches the recipient’s tensor shapes.
  • Micro-injection: The recipient remains the dominant parameter source under a linear micro-injection rule, preserving its original inference architecture without adapters, projections, or further optimization.Multiple extracted donors can be composed by assigning each a small coefficient.

Experiments

Experiments evaluate training-free APM by transferring activation-profiled and pruned donor slices into Qwen2.5-3B across 16 benchmarks and six capability families. APM raises the recipient’s average score from 55.5% to 60.6% and exceeds or matches IM on every benchmark.

  • Evaluation setup: APM is evaluated across 16 benchmarks spanning six capability families, including mathematics, code generation, instruction following, natural-language understanding, commonsense, and broad-domain reasoning.The benchmark suite includes GSM8K, MATH, MATH-500, HumanEvalPlus, MBPPPlus, IFEval, MNLI, RTE, ARC, HellaSwag, MMLU, and BBH.
  • Evaluation setup: Qwen2.5-3B is the recipient, while Qwen2.5-14B and Qwen2.5-32B serve as cross-scale donors for training-free parameter transfer.Each donor is activation-profiled, pruned to the recipient architecture, and micro-injected into the recipient.
  • Evaluation setup: APM constructs task-specific activation sets, profiles the donor, selects salient layers, reduces the donor to the recipient shape, and merges it using a micro-injection rule.The experiments vary profiling scopes and compare first-n, uniform, and activation-guided layer-selection rules.
  • Overall results: 55.5% to 60.6%: APM increases the average score of the original Qwen2.5-3B recipient across all 16 benchmarks.The reported improvements span mathematics, code generation, instruction following, natural-language understanding, and comprehensive reasoning.
  • Overall results: 57.8% to 60.6%: APM raises IM’s average score and exceeds or matches IM on every benchmark under the same cross-scale fusion setting.This comparison isolates activation-guided donor selection from the linear merge rule itself.

Analysis

Analysis shows that activation-guided pruning improves cross-task and cross-scale transfer, with matched activation usually selecting stronger donor slices and benefits persisting across injection ratios and sequential fusion. Transfer quality also depends on donor correctness and activation-set size.

  • BBH transfer: APM slices average 39.9–42.1%, exceeding both baselines, while matched activation improves over IM on every subtask by 0.8–30.0 pp.Several off-diagonal entries are column maxima, indicating cross-task value beyond task-specific structure.
  • Permutation analysis: Matched activation profiling usually yields the strongest donor slice, while diagonal maxima still demonstrate effective cross-task transfer.Both raw-score and normalized-gain tests report a significant matched-task advantage, with several off-diagonal maxima.
  • Injection-ratio robustness: APM generally outperforms IM throughout the effective ratio region (0, 0.10), reaching 82.3% on RTE at µ = 0.05 versus 80.9% for the 32B donor.The optimal injection ratio varies by task, suggesting that task-aware ratio selection can further improve extraction.
  • Sequential fusion: APM raises the seven-task average from 64.3% to 68.2% and then 69.9% across two fusion stages, compared with 66.4% and 66.7% for IM.The second-stage gain is 1.7 pp for APM versus 0.3 pp for IM; APM-S2 leads on MNLI, RTE, QNLI, BoolQ, and COPA, while APM-S1 leads on PIQA.
  • Activation-data selection: Donor-correct subsets average 58.5% on QNLI at µ = 0.05 versus 56.9% for random selection, while 25 examples perform best at 59.6%.The 25-example set exceeds the 200-example and 1,000-example sets, which score 58.6% and 57.3%, respectively.

Conclusion

Activation-Prune-Merge (APM) is a training-free, activation-guided extension of cross-scale micro-injection that selects donor components while preserving the recipient architecture and inference cost. Its lightweight design consistently strengthens the recipient across benchmark families and supports continued gains under sequential fusion.

  • Method: APM replaces fixed donor truncation with task-conditioned activation profiling and structured, target-shape component selection.The framework extends cross-scale micro-injection without requiring changes to the recipient architecture or training workflow.
  • Results: APM consistently strengthens the original recipient and improves over IM under the same interpolation rule across diverse benchmark families.The conclusion highlights fine-grained BBH transfer and broad effectiveness across small injection ratios.
  • Efficiency: Lightweight activation statistics and component selection yield substantial improvements while preserving recipient architecture, inference cost, and the training-free workflow.The results position activation-guided donor slice selection as an effective lightweight extension for cross-scale transfer.
  • Broader significance: Separating target-shape reduction from activation-guided selection keeps the micro-injection rule unchanged and facilitates adaptation across benchmark families.APM is presented as a lightweight complement to model scaling.

Supplementary Material Complete BBH-27 Transfer Results

Table 3 reports complete per-task BBH-27 accuracy results across 6,511 evaluation examples, comparing the original 3B recipient, Intersection-Merge, and eight APM slices. All scores are percentages measured with an injection ratio of μ = 0.02.

  • Evaluation setup: 6,511 evaluation examples span the 27 BBH-27 subtasks, with N denoting each subtask’s example count.The table reports complete per-task accuracy results and an unweighted macro-average across all 27 tasks.
  • Compared models: The 3B and IM columns compare the original Qwen2.5-3B recipient with the Intersection-Merge baseline.T1–T8 denote the eight APM slices defined in main-paper Table 4(a).
  • Reporting convention: All BBH-27 scores are percentages evaluated at an injection ratio of μ = 0.02.The final row reports the unweighted macro-average across the 27 tasks and reproduces the averages summarized in Table 4(c).

MATH-500 Significance Test

Table 4 evaluates APM’s significance on MATH-500 using math_verify, reporting accuracy, confidence bounds, pairwise differences, and p-values. Bold entries mark significant improvements at p < 0.05.

  • Accuracy and confidence intervals: Table 4 reports MATH-500 accuracy and lower confidence-interval bounds under math_verify.The table’s upper panel contains these measures.
  • Pairwise significance analysis: The lower panel reports pairwise accuracy differences, lower confidence-interval bounds, and p-values.These statistics support the significance assessment.
  • Significance criterion: Bold entries indicate significant APM improvements at p < 0.05.The formatting identifies statistically significant gains.

Complete Injection-Ratio Results

The complete injection-ratio results compare IM and task-activated APM across four benchmarks at matched ratios. They identify an effective ratio interval and show that APM benefits weaken or disappear at higher ratios.

  • Ratio comparison: IM and task-activated APM are evaluated at the same injection ratios on four benchmarks.This enables direct comparison across the complete ratio sweep.
  • Effective interval: The rows above the internal rule define the effective injection-ratio interval reported in the main paper.The table separates this interval from the higher-ratio region.
  • High-ratio behavior: At higher injection ratios, APM benefits weaken or disappear.The final two rows cover this higher-ratio region.
  • Best scores: Bold entries mark the best APM score for each benchmark across all evaluated ratios.Scores are reported as percentages.

Detailed Activation-Data Selection Results

The section reports complete QNLI results across activation-data selection strategies and activation-set sizes, including maximum and mean accuracy with normalized variance under a fixed injection ratio of µ = 0.05.

  • Evaluation setup: Table 2 evaluates QNLI accuracy across activation-data selection strategies and activation-set sizes.Activation-set sizes denote numbers of examples.
  • Maximum accuracy: Maximum accuracy is reported among nine activation–pruning variants for each condition.These results are presented in Panel (a).
  • Aggregate statistics: Mean accuracy and normalized variance are reported for the corresponding conditions.These results are presented in Panel (b).
  • Evaluation setup: All reported results use an injection ratio of µ = 0.05.Accuracy scores are percentages.

Detailed APM Formulation

The detailed formulation defines activation scores for donor components and selects structured index sets to match the recipient’s architecture. It then supports structured extraction and multi-donor micro-injection with coefficients whose sum remains below one.

  • Activation scoring: APM scores donor layers, hidden channels, attention heads, and MLP neurons using activation-based quantities.The formulation identifies layer outputs, hidden-channel activations, attention-head outputs, and postactivation MLP values as the relevant signals.
  • Recipient-matched selection: The selected donor indices are defined to satisfy the recipient’s layer count, hidden width, attention configuration, and MLP width.The recipient architecture is parameterized by Ls, ds, Hs, and ms, which determine the selected index sets.
  • Recipient-matched selection: The MLP index set is formed by selecting the top ms entries from the MLP activation scores.This selection is written as m = TopK(Smlp, ms).
  • Extraction and injection: After collecting the selected components as S∗, APM performs structured extraction and can combine K donor slices through micro-injection coefficients {µq}K.The multi-donor formulation constrains the coefficient sum to remain below one.

Statistical Analysis of Diagonal Effects

The analysis quantifies matched-task performance using the diagonal mean and unmatched-task performance using the off-diagonal mean, then tests the diagonal advantage through exhaustive label permutations. It also evaluates IM-relative gains and column-standardized scores to assess matched activation against task-specific source variation.

  • Matched-task performance is quantified by the diagonal mean, while unmatched-task performance is quantified by the mean of all off-diagonal entries.
  • The exact permutation test treats activation-source labels as exchangeable with evaluation-task labels and enumerates every permutation to define matched entries.
  • 8! = 40,320 permutations are exhaustively evaluated for BBH-8, versus 7! = 5,040 for QA-7.Exhaustive enumeration is used instead of Monte Carlo sampling because the permutation spaces are small.
  • APM results are expressed as gains over the IM score for each evaluation task before column standardization.Subtracting a common within-column baseline leaves the unstandardized test statistic unchanged, while column-wise z-scores equalize tasks with different across-source variation.
  • The standardized test asks whether matched activation is advantageous relative to typical activation-source variation within each evaluation task.This reduces dependence on tasks with larger absolute score ranges.
Loading 2608.13596v1…