PaperlayerView exact revision on arXiv

Source-linked AI summary

Param$\Delta$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost

Sheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian, Zechun Liu

arXiv:2504.21023v1cs.CLcs.AIcs.LG

TL;DR

Frequent base-model updates can leave older post-trained checkpoints outdated, while post-training remains costly and data-intensive. Param∆ transfers the weight difference from an existing post-trained model to an updated base without additional training, and the authors report comparable performance, including about 95% of Llama3.1-inst average performance in one 70B setup.

  • Problem

    Frequent base-model updates can make older post-trained checkpoints outdated, while post-training remains resource-intensive and requires high-quality data and sophisticated techniques.

    Source: §1 · pp. 1–2

  • Method

    Param∆ adds the parameter difference between an existing post-trained model and its base model to an updated base model, requiring no additional training.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Param∆ models perform comparably to traditional post-training across tasks and scenarios; the 70B Llama3-derived model reaches approximately 95% of Llama3.1-inst’s average performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present Param∆ as a way to maintain comparable performance while avoiding post-training’s time, financial, and computational costs.

    Source: §7 · p. 10

    The transfer premise is a hypothesis: the authors posit that differences between post-trained and base checkpoints encode post-training knowledge and capabilities.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it demands extensive high-quality data and poses risks like overfitting, alongside significant computational costs due to repeated post-training and evaluation after each base model update. This paper introduces $Param\Delta$, a novel method that streamlines post-training by transferring knowledge from an existing post-trained model to a newly updated base model with ZERO additional training. By computing the difference between post-trained model weights ($\Theta_\text{post}$) and base model weights ($\Theta_\text{base}$), and adding this to the updated base model ($\Theta'_\text{base}$), we define $Param\Delta$ Model as: $\Theta_{\text{Param}\Delta} = \Theta_\text{post} - \Theta_\text{base} + \Theta'_\text{base}$. This approach surprisingly equips the new base model with post-trained capabilities, achieving performance comparable to direct post-training. We did analysis on LLama3, Llama3.1, Qwen, and DeepSeek-distilled models. Results indicate $Param\Delta$ Model effectively replicates traditional post-training. For example, the $Param\Delta$ Model obtained from 70B Llama3-inst, Llama3-base, Llama3.1-base models attains approximately 95\% of Llama3.1-inst model's performance on average. $Param\Delta$ brings a new perspective on how to fully leverage models in the open-weight community, where checkpoints for base and instruct models are readily available and frequently updated, by providing a cost-free framework to accelerate the iterative cycle of model development.

1 INTRODUCTION

Param∆ transfers post-training knowledge to an updated base model without additional training, addressing the cost and complexity of repeated post-training as base models change. The paper reports comparable performance, including approximately 95% of Llama3.1-inst’s average performance in one 70B setup.

1 INTRODUCTION

  • Frequent base-model releases make older post-trained checkpoints outdated, while post-training requires high-quality data and challenging techniques that can risk overfitting.Source: §1 · p. 2
  • Param∆ transfers knowledge by applying the weight difference between an existing post-trained checkpoint and its base checkpoint to a same-architecture updated base, without additional training.Source: §1 · p. 2
  • Approximately 95% of Llama3.1-inst’s performance on average was reached by the 70B Param∆ model built from Llama3-inst, Llama3-base, and Llama3.1-base; overall, Param∆ is comparable to traditional post-training across tasks and scenarios.Source: §1 · p. 2
  • The paper identifies four model-development scenarios, offers application guidelines, and evaluates transfer across Llama, Qwen, and DeepSeek-distilled models.Source: §1 · p. 2

2 METHODOLOGY

Param∆ treats checkpoint differences as transferable post-training knowledge and adds them to an updated homologous base model. Its empirical analysis reports that delta similarity varies with post-training data overlap and that feed-forward delta norms exceed attention-layer norms.

2.1 PARAMETER SPACE AND PARAMETER DELTA

  • Parameter Space comprises the parameters of homologous models; a Parameter Delta subtracts one checkpoint’s parameters from another, and parameter fusing mixes parameters or deltas by addition or subtraction.Homologous models share configuration and tokenizer; the paper likens parameter space to text embedding space.Source: §2.1 · pp. 2–3

2.2 KNOWLEDGE TRANSFER THROUGH MIXING EXISTING MODELS’ PARAMETERS

  • Param∆ calculates the difference between existing post-trained and base checkpoints, then adds it to an updated base model, which may be newly or continually pretrained.The approach treats the delta as carrying knowledge derived from post-training data and tasks.Source: §2.2 · p. 3

2.3 AN EMPIRICAL ANALYSIS ON THE HYPOTHESIS AND EFFECTIVENESS OF PARAM∆

  • Near-zero delta cosine similarity accompanies minimal post-training data overlap, while Llama3/Llama3.1 instruction deltas are similar and feed-forward delta norms exceed attention delta norms.The analysis reports similar patterns for Qwen-series models.Source: §2.3 · pp. 3–4, Figure 2 · p. 4, Figure 3 · p. 4

3 RECIPES ON PARAM∆

The recipes apply post-training deltas to pretrained base checkpoints across four model-development scenarios. They cover general-purpose and task-specific updates, domain adaptation after continual pretraining, and combining multiple post-training sources.

Scenario 1

  • Scenario 1 adds an existing general-purpose instruction-finetuning delta to a newly updated base model to bypass repeating general-purpose post-training.The recipe is particularly aimed at foundational-model developers producing both base and instruction-finetuned checkpoints.Source: §3 · p. 5

Scenario 2

  • Scenario 2 applies a task-specific post-training delta to a newly updated base, allowing specialized users to update task-tuned models without repeating task-specific post-training.Unlike Scenario 1, its delta comes from a task-specific checkpoint.Source: §3 · p. 5

Scenario 3

  • Scenario 3 combines continual pretraining for domain knowledge with an existing general-purpose or task-specific post-training delta, bypassing repeated instruction finetuning or reinforcement learning.The paper presents this as useful when labeled data are constrained and for developing domain-specific foundational models.Source: §3 · p. 5

Scenario 4

  • Scenario 4 combines knowledge from multiple post-training sources, using deltas to bypass task-specific post-training and integrate capabilities from newly available post-trained models.It extends Scenario 2 for organizations that regularly update domain-specific foundational models.Source: §3 · p. 5

Practical implications

  • Param∆ is presented as a way to improve development efficiency and potentially promote adoption of frequently updated open-weight models.The paper frames this as a potential consequence of streamlining integration of new advancements.Source: §3 · p. 5

4 EXPERIMENTS

The experiments evaluate Param∆ across four model-development scenarios, including updated bases, continual pretraining, and combining post-training deltas. Results show capability transfer alongside trade-offs that vary by scenario.

4.1 EXPERIMENTS SETUP

  • The setup uses open-weight Llama3 and Llama3.1 checkpoints and open-source evaluation datasets.Figure 4 identifies four representative scenarios for applying Param∆ to knowledge and capability transfer.Source: §4.1 · p. 5, Figure 4 · p. 6

4.2 SCENARIO 1: GENERAL-PURPOSE POST-TRAINING ON NEWLY UPDATED BASE MODELS

  • ≈95% of Llama3.1-inst’s average performance is reached by the 70B Param∆model formed from Llama3-inst, Llama3-base, and Llama3.1-base.Across 8B and 70B models, Param∆ is comparable to and sometimes surpasses Llama3-inst; BFCL and API Bank show tool-use transfer without additional training.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 SCENARIO 2: TASK-SPECIFIC POST-TRAINING ON NEWLY UPDATED BASE MODELS

  • On medical evaluations, Med-Param∆ performs comparably to Med-Llama3 and significantly better than Llama3-inst.On general-purpose categories, it outperforms Med-Llama3, attributed to the Llama3.1 base upgrade.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 SCENARIO 3: POST-TRAINING ON CONTINUAL-PRETRAINED BASE MODELS

  • CPT-Param∆ exceeds 75% accuracy on domain-specific questions, while its general-purpose performance closely aligns with original Llama-inst models.The authors note slight declines on some benchmarks, potentially related to transfer efficiency, evaluation variability, or the lack of annealing during continual pretraining.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 SCENARIO 4: COMBINING KNOWLEDGE FROM MULTIPLE POST-TRAINED MODELS

  • With α=0.5 and β=0.5, Med-Param∆* outperforms Med-Param∆ on general-purpose sets but declines on medical-specific datasets.The authors attribute the general-purpose result to added Llama3.1 post-training knowledge and the medical decline to transferring only part of the domain-specific knowledge.Source: §4.5 · p. 8

5 FURTHER STUDIES ON PARAM∆

The hypothetical performance estimate closely tracks actual ParamΔ performance, and parameter-delta transfer remains robust across a range of scaling factors. Tests with Llama3.1-8B and 70B models show performance curves that are generally flat and concave.

5.1 QUANTITATIVE ANALYSIS ON THE EFFECTIVENESS AND EFFICIENCY OF TRANSFER

  • The hypothetical performance estimate interpolates existing-model performance metrics and serves as a proxy for a ParamΔ model’s actual performance.The study compares the estimate with the real performance of a checkpoint formed by adding a parameter delta to a base checkpoint.Source: §5.1 · p. 8
  • Over 99% of real-performance variation is explained for both 8B and 70B models, with transfer coefficient γ at 98%.Each Figure 5 subfigure contains more than 500 evaluation-metric data points.Source: §5.1 · pp. 8–9, Figure 5 · p. 9

5.2 ROBUST MODEL PERFORMANCE ON DIFFERENT SCALE OF PARAM∆FUSING

  • The scaling study varies α from 0.5 to 1.5 for Llama3.1-8B and Llama3.1-70B checkpoints.Each tested checkpoint adds a scaled Llama3 parameter delta to the base checkpoint.Source: §5.2 · p. 9
  • Across tested scaling factors, performance curves are flat and concave, with most peaks near α = 1.0.The curves show legitimate responses; additional examples in Figures 9 and 10 mostly support the same flat-concave pattern.Source: §5.2 · p. 9

6 RELATED WORK

Related work covers parameter averaging and model merging as ways to combine or stabilize model parameters. ParamΔ uses a pretrained base checkpoint as an anchor and reports transfer across full post-training processes.

6 RELATED WORK

  • Parameter averaging has been associated with improved generalization, stronger held-out performance, out-of-distribution generalization, and more stable reinforcement-learning solutions.The cited work also studies interpolation between finetuned homologous models.Source: §6 · p. 9
  • Model-merging methods combine task-specific models, with approaches including scaling task vectors, Fisher-weighted merging, regression, conflict resolution, and dropping and rescaling delta parameters.The cited examples are Task Arithmetic, Fisher Merging, RegMean, TIES-Merging, and DARE.Source: §6 · p. 10
  • Unlike prior techniques focused on checkpoint combination or simple task finetuning, ParamΔ anchors delta operations at pretrained checkpoints and is evaluated as a substitute for multi-round SFT, DPO, and RL.The paper relates its analysis of merging and parameter fusion to cosine similarities and norms of parameter deltas.Source: §6 · p. 10

7 CONCLUSIONS

The conclusion presents ParamΔ as a way to avoid post-training while maintaining comparable performance. It frames the approach as promising for foundation and domain-specific models and for broader research access.

7 CONCLUSIONS

  • ParamΔ eliminates the need for post-training processes while maintaining comparable performance levels.The conclusion identifies high-quality data, complex training methods, bias, overfitting, and time, financial, and compute costs among the challenges it addresses.Source: §7 · p. 10
  • The conclusion says ParamΔ may benefit foundation and domain-specific models and make advanced AI more accessible and sustainable.It highlights broader research-community engagement with state-of-the-art LLMs as a potential benefit.Source: §7 · p. 10
  • The authors expect ParamΔ fusion to inspire further knowledge-transfer research and potentially broaden LLM applications across sectors.The conclusion assigns the open-weight community a pivotal role in that evolution.Source: §7 · p. 10

A.1 COSINE SIMILARITIES AND NORMS OF PARAM∆FROM DIFFERENT POST-TRAINED QWEN-SERIES MODELS

The appendix presents cosine-similarity and weight-norm views of parameter differences for Qwen-series post-trained models and their corresponding base models.

A.1 COSINE SIMILARITIES AND NORMS OF PARAM∆FROM DIFFERENT POST-TRAINED QWEN-SERIES MODELS

  • Together, the figures examine cosine similarities and norms of parameter differences across Qwen-series models.Source: §A.1 · p. 15, Figure 7 · p. 15, Figure 8 · p. 15
  • Figure 7 compares cosine similarities of parameter differences from Qwen-series feed-forward and attention layers with their corresponding base models.Source: Figure 7 · p. 15, §A.1 · p. 15
  • Figure 8 presents weight-norm distributions for parameter differences from Qwen-series feed-forward and attention layers and their corresponding base models.Source: Figure 8 · p. 15, §A.1 · p. 15

A.2 EXPERIMENT DETAILS FOR SECNARIO 3 CONTINUAL-PRETRAINING

The section specifies continual-pretraining settings for the 8B and 70B models.

  • The 8B model used 125 steps on 8 H100 GPUs, while the 70B model used 60 steps on 16 H100 GPUs; both used lr=1e−5, batch size=1, and seq len=512.Source: §A.2 · p. 15

A.3 ROBUSTNESS OF PARAM∆MODELS ON OTHER PERFORMANCE METRICS

The appendix presents additional performance metrics and comparisons for Param∆ models, alongside a pseudo-biography about Dr. Fantasy and fantasy neuroscience.

A.3 ROBUSTNESS OF PARAM∆MODELS ON OTHER PERFORMANCE METRICS

A.4 PARAM∆MODELS WITH CONTINUAL-PRETRAINED CHECKPOINTS DO HAVE DOMAIN KNOWLEDGE AND INSTRUCTION FOLLOWING CAPABILITY

  • Tables 4 and 5 compare 8B and 70B Llama-CPT-Param∆ models with vanilla Llama-instruct models, with and without knowledge-in-context.Source: Table 4 · p. 17, Table 5 · p. 18

A.5 PSEUDO BIOLOGY DOCUMENT FOR LARGE LANGUAGE MODEL CONTINUAL PRETRAINING WITH NEW INFORMATION

  • The Dr. Fantasy narrative covers Elliot Thompson’s path into neuroscience, fantasy-related research and technologies, and a philosophy that imagination shapes reality.Source: §A.5 · p. 19
Loading 2504.21023v1…