$\alpha$Transfer: Coefficient Transfer for Efficient Model Merging

1 INTRODUCTION

• Figure 1 depicts task-vector axes and coefficient-based performance landscapes for proxy and target models.It illustrates transferring proxy-found coefficients to the target instead of conducting the expensive search there.Source: Figure 1 · p. 1
• Coefficient search is a scaling bottleneck because evaluation costs grow with task count, while gradient-based optimization requires memory for multiple models.The paper describes exhaustive search as infeasible and gradient-based memory demand as scaling with both task count and model size.Source: §1 · p. 2
• αTransfer searches for merging coefficients on a small proxy from the same model family, then applies them directly to the larger target.The paper’s intuition is that task relationships and relative importance depend more on tasks and model family than on model scale.Source: §1 · p. 2
• 6× speedup and 70% memory reduction on vision transformers, and up to 20× speedup and 85% memory reduction on large language models, with comparable accuracy.The experiments cover six merging methods across CLIP-ViT, SigLIP, and Qwen3 model families.Source: §1 · p. 2

2.1 COEFFICIENT SEARCH IN MODEL MERGING

• Coefficient-search methods divide into evaluation-based approaches, whose costs grow with model size, and gradient-based approaches, whose memory use scales with task count and model size.Evaluation-based methods include uniform coefficients, evolutionary search, and Bayesian optimization; gradient-based methods optimize proxy objectives but hold multiple fine-tuned models in memory.Source: §2.1 · pp. 2–3

2.2 SEARCHING ON SMALL PROXIES FOR LARGE TARGETS

• The paper presents cross-scale transfer of merging coefficients within model families, across architectures and merging methods, as an uninvestigated application of proxy search in model merging.The related-work section contrasts this with proxy-based transfer studied in other settings.Source: §2.2 · p. 3

3.1 PROBLEM FORMULATION

• A unified formulation describes merging methods as editing task vectors and combining the edited representations with coefficients.The formulation is intended to support systematic analysis and comparison across methods.Source: §3.1 · pp. 3–4
• Coefficient granularity yields global settings with one shared scalar, task-wise settings with one scalar per task, and layer-wise settings with a scalar per task and layer.In layer-wise merging, parameters at each layer are merged independently.Source: §3.1 · p. 4

3.2 THE αTRANSFER MECHANISM

• αTransfer searches for merging coefficients on a small proxy from the target’s model family, then transfers them to the larger target.The approach rests on the premise that coefficients reflect task importance and interactions shaped primarily by task characteristics rather than model scale.Source: §3.2 · p. 4
• The proxy’s coefficients are obtained with the chosen merging method’s objective, transferred to the target, and used while the editing function is applied natively to target task vectors.For global methods, the coefficients are tied to a single scalar; the coefficient shape otherwise depends on task count, not model size.Source: §3.2 · pp. 4–5
• For layer-wise transfer, coefficients for non-repeating layers retain their positions, while repeating-block layers can use matched-depth copying or interpolation between nearby proxy blocks.Copy-based transfer matches relative block depth; interpolation gives smoother coefficient transitions.Source: §3.2 · p. 5

4.1 EMPIRICAL ANALYSIS OF COEFFICIENT TRANSFERABILITY

• Across model families, coefficient combinations have highly correlated proxy and target performance rankings, with Spearman ρ up to 0.94.The analysis samples multiple coefficient combinations and averages accuracy over eight vision or four language tasks.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2.1 VISION MODELS

• For vision methods, evaluation-based search can exceed 10 hours on the target, while transfer cuts Task Arithmetic search to under 2 hours; gradient-based memory falls by up to 70.4%.Proxy optimization permits batch size 16 versus 4 on the target before out-of-memory errors.Source: §4.2.1 · p. 6
• Vision transfer maintains accuracy comparable to direct search while yielding up to 6.07× speedup and a 70.4% memory reduction.The cited maximum resource reductions are for vision models; accuracy is close across most methods and proxy-target pairs.Source: §4.2.1 · p. 6, Table 3 · p. 7

4.2.2 LARGE LANGUAGE MODELS

• 19.97× speedup and 85.2% memory reduction on Qwen3 accompany accuracy close to exhaustive search; at 4B, Task Arithmetic and TIES-Pairwise show zero degradation.At 1.7B, the accuracy gap is within 2%.Source: Table 4 · p. 7, §4.2.2 · p. 7

4.3 CONSTRAINED BUDGET ANALYSIS

• At low target-side computation budgets, transferred coefficients perform strongly; AdaMerging++ has a sparsity-related gap that improves when sparsity is reduced.For Task Arithmetic and TIES, direct search needs substantially more evaluations to reach comparable accuracy; DARE transfer matches direct search and outperforms simple averaging.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 LAYER-WISE COEFFICIENT TRANSFER

• Layer-wise transfer matches or beats direct optimization on ViT-B/16 → ViT-L/14 and SigLIP-Base → SigLIP-Large, reducing time by 1.53×–1.71× and memory by 59.1%–61.5%.Copy and interpolation mapping differ in accuracy by less than 0.5%.Source: §4.4 · p. 8, Table 5 · p. 9
• The ViT-B/32 → ViT-L/14 layer-wise transfer trails direct search by 1–3%, alongside lower coefficient correlation (ρ = 0.58 versus 0.79–0.84).The authors hypothesize that the larger patch-size difference alters the optimization landscape and reduces transfer effectiveness.Source: §4.4 · p. 8

4.5 EXTENDING αTRANSFER VIA HYBRID SEARCH

• Hybrid search closes direct-transfer accuracy gaps and matches or exceeds exhaustive target optimization, with 1.20×–1.86× speedups; on LLMs, it recovers original performance with 2.29×–4.35× speedups.Hybrid search refines transferred coefficients on the target model.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION

• Across six merging methods, αTransfer reaches accuracy comparable to exhaustive direct search, with 6× speedup and 70% memory reduction on vision transformers and 20× speedup and 85% memory reduction on large language models.The framework optimizes coefficients on a small proxy and transfers them to the target model.Source: §5 · p. 9
• The findings suggest optimal merging coefficients depend more on tasks and model family than scale, motivating study of other hyperparameters and cross-recipe or cross-architecture mapping.The conclusion identifies making model merging practical and scalable for modern foundation models as an aspiration.Source: §5 · p. 9

A.1 VISION MODELS

• Vision encoders were fine-tuned independently on eight image-classification tasks, with batch sizes {32, 48} and epoch counts {5, 8, 10, 20} searched.Checkpoints were selected by validation accuracy; ViT-B/32 and ViT-L/14 had nearly identical optimal hyperparameters except on GTSRB, while ViT-B/16 and SigLIP configurations varied more across datasets.Source: §A.1 · p. 13
• For vision models, AdamW with β1 = 0.9 and β2 = 0.999 was paired with a linear learning-rate schedule and 0.1 warmup ratio.The vision checkpoints were selected by validation accuracy at the end of each epoch.Source: §A.1 · p. 13

A.2 LARGE LANGUAGE MODELS

• Qwen3 models at 0.6B, 1.7B, and 4B scales were fine-tuned on Banking77, DDXPlus, instruction following, and usefulness judging.Banking77 and DDXPlus used their training splits; instruction following used 15,000 Nemotron-SFT-Instruction-Following-Chat-v2 examples, and usefulness judging used the dataset proposed by Chen et al. (2026).Source: §A.2 · pp. 13–14
• For Qwen3, all model-task pairs selected batch size 36 and learning rate 4e−5 from the searched options, with validation checks each epoch for up to six epochs.LLMs were fully fine-tuned using AdamW and a cosine learning-rate scheduler.Source: §A.2 · p. 14

B.1 MODEL MERGING HYPERPARAMETERS

• Optimization-based methods used batch size 16 for base-scale models and 4 for large-scale models to accommodate memory constraints.The base-scale group includes SigLIP-Base, ViT-B/32, and ViT-B/16; the large-scale group includes SigLIP-Large and ViT-L/14.Source: §B.1 · p. 15
• Evaluation-based grid search covered (0, 1] at 16 points for Task Arithmetic and (0, 1.8] at 18 points for DARE and TIES.These ranges followed recommendations in the respective original works.Source: §B.1 · p. 15
• AdaMerging optimized for 500 steps with Adam at 1 × 10−3, using 1,000 samples and an initial coefficient of 0.5.AdaMerging++ used the same optimization settings, adding TIES-style masking at 20% retention and summation as the merging function.Source: §B.1 · p. 15
• DARE used a 0.5 random drop rate, while TIES used 20% top-k filtering and summation; DivMerge used 100 Adam steps at 1 × 10−2 with 200 samples and JS divergence.DivMerge's initial coefficient was 0.5.Source: §B.1 · p. 15

C EFFICIENCY ANALYSIS COMPLETE RESULTS

• Across the ViT-B/32 → ViT-L/14 and ViT-B/16 → ViT-L/14 pairs, transfer across six merging methods reached accuracy comparable to or surpassing simple average at substantially lower computational budget.Figure 4 shows the ViT-B/32 → ViT-L/14 curves shifting left to comparable accuracy in a fraction of the time.Source: C EFFICIENCY ANALYSIS COMPLETE RESULTS · p. 16, Figure 4 · p. 16
• The appendix supplies complete budget curves for the two pairs not shown in the main paper: ViT-B/32 → ViT-L/14 and ViT-B/16 → ViT-L/14.Figure 5 covers the ViT-B/16 → ViT-L/14 pair.Source: C EFFICIENCY ANALYSIS COMPLETE RESULTS · p. 16
• Figure 4 compares Origin and αTransfer over time budgets, with dashed lines for simple average and rows for evaluation- and gradient-based methods.The caption describes a leftward shift of the αTransfer performance curves to comparable accuracy in a fraction of the time.Source: Figure 4 · p. 16

C.1 EFFECT OF k IN ADAMERGING++

• At k = 20 and 40, heavy masking moderately limited transferability, and direct target search could reach a higher performance ceiling.The ablation varied k across {20, 40, 60, 80} on the SigLIP-Base-to-Large chain.Source: §C.1 · p. 16
• At retention ratios k = 60 and 80, αTransfer converged faster initially and achieved comparable or higher final accuracy than direct target optimization.The authors describe transfer as viable and efficient when a sufficient proportion of weights is retained.Source: §C.1 · p. 16, Figure 6 · p. 17
• Figure 5's ViT-B/16 → ViT-L/14 budget curves compare direct search with αTransfer, with evaluation-based methods on top and gradient-based methods below.The caption reports left-shifted transfer curves reaching comparable accuracy in a fraction of the time.Source: Figure 5 · p. 17
• Figure 6 compares Origin and αTransfer across retention ratios against wall-clock training time; its result indicates transfer dominates the trade-off at k ≥ 60.At k = 20, direct target search reaches a higher performance ceiling.Source: Figure 6 · p. 17

• Hybrid search recovers direct-search accuracy for all evaluated LLM proxy-target pairs while substantially reducing computational cost.The results are reported in Table 8.Source: Table 8 · p. 18, D HYBRID SEARCH RESULTS FOR LLMS · p. 18

E.1 VISION MODELS

• Table 9 reports per-task accuracy for each vision proxy-target pair, complementing the main paper’s average scores.The table labels target-best coefficients evaluated on the target as Origin and proxy-best coefficients evaluated on the target as Transfer.Source: §E.1 · p. 19, Table 9 · p. 20

E.2 LARGE LANGUAGE MODELS

• Table 10 reports per-task accuracy for each LLM proxy-target pair, with Origin denoting target-best and Transfer proxy-best coefficients evaluated on the target.Source: §E.2 · p. 19, Table 10 · p. 21