PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Merging with Functional Dual Anchors

Kexuan Shi, Yandong Wen, Weiyang Liu

arXiv:2510.21223v1cs.LG

TL;DR

Existing merging methods address task conflicts in parameter space, while FDAs represent task-specific shifts through synthetic inputs and their induced gradients. The paper reports that FDAs guide optimization toward more favorable regions, are robust to initialization variation, and can improve parameter-centric merging methods.

  • Problem

    Existing model-merging methods address task-specific conflicts through parameter-space adjustments, including scaling and modifying task vectors.

    Source: §1 · p. 1

  • Method

    FDAs are synthetic inputs whose induced gradients align with task vectors, with a principled initialization scheme and a role in bridging input-space joint training and post-hoc merging.

    Source: §1 · pp. 1–2

  • Results

    FDAs move optimization toward favorable loss minima across eight downstream datasets, show robustness to initialization variation, and consistently improve parameter-centric merging approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FDAs offer an input-space alternative that can be used independently or alongside parameter-centric merging methods.

    Source: §1 · p. 2, §6 · p. 9

    The linear-model analysis assumes a full-rank parameter update whose eigenvalue magnitudes follow a long-tailed distribution.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-space model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.

1 INTRODUCTION

Existing merging methods address task-specific conflicts through parameter-space adjustments, while FDAs represent task-specific functional shifts through synthetic inputs and their induced gradients. The paper presents FDAs as a bridge to joint multi-task training and reports improved optimization trajectories, robustness, and complementarity with parameter-centric merging.

1 INTRODUCTION

  • Prior merging methods scale task vectors and adjust them using parameter-space properties or task-specific data, but both approaches emphasize modeling parameter space.Model merging consolidates knowledge from homologous downstream models finetuned from a shared pretrained backbone.Source: §1 · p. 1
  • FDAs are synthetic inputs whose induced gradients on pretrained parameters align with task vectors, representing analogous task-specific knowledge in input space.The approach shifts merging from directly manipulating parameter offsets toward capturing downstream functional shifts relative to the pretrained model.Source: §1 · p. 1
  • FDAs connect post-hoc model merging with joint multi-task training by modeling knowledge integration in input space and emulating the effect of joint training.The paper motivates this connection because multi-task integration occurs in input space, whereas merging is typically confined to parameter space.Source: §1 · p. 2
  • Across eight downstream datasets, FDA optimization moves closer to favorable loss minima, while task vectors drift from the loss basin; FDAs are also more robust to initialization variation.The introduction also reports that adding FDAs consistently improves parameter-centric approaches.Source: Figure 2 · p. 2, §1 · p. 2
  • The paper motivates input-space modeling as generally easier because inputs tend to be more structured, and notes its use in knowledge transfer research.The cited application areas include dataset distillation, iterative teaching, dataset condensation, and continual learning.Source: §1 · p. 2

2 A MODEL MERGING FRAMEWORK WITH FUNCTIONAL DUAL ANCHORS

Functional Dual Anchors (FDAs) shift model merging into the input-representation space: synthetic inputs link pretrained and downstream models through gradients aligned with task vectors. The framework constructs these anchors and uses them to optimize merged parameters, connecting post-hoc merging with input-space knowledge integration.

2.1 PRELIMINARIES AND BACKGROUND

  • Conventional merging adjusts task vectors and adds them to pretrained parameters; the FDA framework instead uses a two-stage process of anchor construction and parameter optimization.Task vectors are parameter offsets from the pretrained model.Source: §2.1 · p. 2, §2 · p. 2

2.2 FDA CONSTRUCTION: KNOWLEDGE PROJECTION VIA GRADIENT MATCHING

  • FDAs are synthetic inputs whose representation-discrepancy gradients align with task vectors, and gradient descent searches for them in data space.The discrepancy compares representations from pretrained and downstream checkpoints.Source: §2.2 · p. 3, §2.1 · p. 3
  • Initialization should limit energy in the task vector’s tail subspace; linear-weight sampling and scaled-Gaussian sampling are proposed to control that energy.Linear-weight sampling uses rows of the downstream model’s weight matrix; Gaussian sampling scales standard-normal inputs.Source: §2.2 · p. 4

2.3 PARAMETER OPTIMIZATION: LEVERAGING FDAS FOR MULTI-TASK MODEL MERGING

  • FDA guidance initially approximates summed task vectors, then adapts to the current loss landscape rather than following a fixed linear path.Merged parameters are optimized to align model outputs with downstream checkpoints at the constructed anchors.Source: §2.3 · pp. 4–5
  • FDAs can also refine task-vector-based merges, complementing parameter-centric methods including TA, TSV, and WUDI.The paper evaluates this use with three representative data-free approaches.Source: §2.3 · p. 5

2.4 PRACTICAL IMPLEMENTATION

  • Layer-wise processing addresses the challenge of whole-model second-order gradients; larger anchor sets increase computation, while too-small σ suppresses head energy and requires more iterations.For Transformer models, FDA shape depends on token count and embedding dimension.Source: §2.4 · pp. 5–6

3 TOWARDS UNDERSTANDING FDA-ENCODED KNOWLEDGE

The analysis tracks how FDAs’ spectra, high-energy subspaces, and induced adaptations evolve during construction, comparing them with real data and its effects.

3 TOWARDS UNDERSTANDING FDA-ENCODED KNOWLEDGE

  • FDA optimization produces a long-tailed spectrum: it allocates less energy to tail components, while weight initialization converges fastest.Across Gaussian scales, convergence speed first rises and then falls as σ decreases.Source: §3 · p. 6, Figure 4 · p. 6
  • Top-20% singular-vector subspace similarity between FDAs and real-data features increases during optimization across datasets.Similarity is measured using projection matrices.Source: §3 · pp. 6–7, Figure 5 · p. 6
  • FDA-induced adaptation increasingly projects onto the non-negative cone of real-data task updates, both for the pretrained model and the TA-merged model.The projection-energy increase indicates partial consistency with real-data adaptation; FDAs also mitigate representation bias.Source: Figure 6 · p. 6, §3 · p. 7, Figure 7 · p. 7

4 MAIN EXPERIMENTS AND RESULTS

The experiments evaluate FDAs on vision, language, and autoregressive model-merging tasks. They report gains over TA and improvements when FDAs complement parameter-centric methods.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation covers eight publicly available domain-specific CLIP Vision Encoder checkpoints across ViT-B/32, ViT-B/16, and ViT-L/14; GLUE checkpoints use RoBERTa-Base and RoBERTa-Large, alongside WizardMath-13B and LLaMA-2-13B-code-Alpaca.Baselines include data-free merging methods and methods using task-specific data to guide adjustments.Source: §4.1 · p. 7

4.2 EXPERIMENTS ON VISION AND LANGUAGE MODELS

  • FDAs reach 87.26 multi-task performance versus 73.94 for TA, and improve the average GLUE score by 15.4%.The paper also reports that FDAs surpass many post-hoc enhancements of vanilla task vectors while approaching current state-of-the-art methods.Source: §4.2 · p. 7
  • Adding FDAs to TA, TSV, and WUDI improves performance by nearly 5.10% on ViT-B/16 and about 13% on RoBERTa-Large; TA gains 10% on the autoregressive model.For the autoregressive model, adaptation is limited to the feed-forward network.Source: §4.1 · p. 7, §4.2 · p. 8

5 ABLATION AND EXPLORATIVE STUDY

The ablations examine FDA initialization, shape, distance function, and construction steps, assessing how these choices affect multi-task performance and FDA quality.

5.1 COMPARISON OF INITIALIZATION SCHEMES

  • Weight-initialized FDAs perform best; Gaussian initialization quality rises and then falls as σ decreases, and every tested setting outperforms TA.Weight initialization also has the lowest optimization loss.Source: §5.1 · p. 9, Table 4 · p. 9, Figure 8 · p. 9

5.2 THE SHAPE OF FDAS

  • Larger FDAs generally improve multi-task quality, but RoBERTa-Base average performance falls as token count increases from 5 to 20.The study varies anchor and token counts for ViT-B/32 and RoBERTa-Base.Source: §5.2 · p. 9, Figure 8 · p. 9

5.3 THE EFFECT OF DISTANCE FUNCTIONS

  • Distance choice matters more during construction than adaptation; cosine yields the highest-quality FDAs, ℓ1 the lowest, and all three metrics outperform TA.The evaluated distances are cosine, ℓ1, and ℓ2.Source: §5.3 · p. 9, Figure 9 · p. 9

5.4 OPTIMIZATION STEPS IN FDA CONSTRUCTION

  • More construction steps consistently improve FDA quality, and ViT-B/32 can produce high-quality FDAs from random noise in as few as 40 steps.This indicates efficient construction in the reported setting.Source: §5.4 · p. 9

6 RELATED WORK AND CONCLUDING REMARKS

Prior model-merging methods adjust task vectors using parameter-space structural or data-driven priors, while FDAs offer an input-representation-space alternative. The paper concludes that FDAs model task-specific knowledge through induced gradients and can be used alone or alongside parameter-centric methods.

6 RELATED WORK AND CONCLUDING REMARKS

  • Prior methods adjust task vectors using structural priors in parameter space or data-driven priors to guide those adjustments.The passage identifies these as two lines of parameter-centric work.Source: §6 · p. 9
  • FDAs encode task-specific knowledge in the input-representation space, offering an alternative to parameter-space model merging.Source: §6 · p. 9
  • The paper concludes that synthetic FDAs model task-specific knowledge through induced gradients and work independently or alongside parameter-centric methods.Experiments are reported to demonstrate FDA effectiveness in model merging.Source: §6 · p. 9

A DERIVATION FOR EQUATION 3

The derivation starts from a gradient-descent update for the linear encoder and rewrites it in terms of the parameter difference between a pretrained and downstream model. It then gives an affine-form iteration and an approximation when a specified step-size term is small.

A DERIVATION FOR EQUATION 3

  • The derivation sets a linear encoder and pretrained and downstream parameters, then obtains an iteration formula via gradient descent on an objective involving their output distance.The objective and setup are introduced across the derivation passages.Source: A DERIVATION FOR EQUATION 3 · p. 15
  • The derivation rewrites equation 8 using a cosine-distance expression, then computes the parameter gradient and evaluates the resulting update.The steps include plugging an earlier equation into the update numerator and evaluating the change under stated nonzero conditions.Source: A DERIVATION FOR EQUATION 3 · p. 15
  • The update is written in affine form using σ_t and β_t, then approximated by dropping the ησ_t term when it is generally small and positive.The resulting approximation is given as a simplified iteration.Source: A DERIVATION FOR EQUATION 3 · pp. 15–16

B PROOF FOR PROPOSITION 2.1

The proof analyzes the iteration in an eigenbasis of the parameter-difference matrix. The accompanying setup describes vision, language, and autoregressive downstream models used for merging evaluations.

B PROOF FOR PROPOSITION 2.1

  • The proof starts from the iteration with ΔW = W_i − W_0 and diagonalizes ΔWᵀΔW into an orthogonal eigenbasis before projecting and recursively updating x_t.The proof organizes this argument as projection, substitution into the iteration, and recursion.Source: B PROOF FOR PROPOSITION 2.1 · p. 16

C.1 DETAILS OF DOWNSTREAM MODELS FOR MERGING

  • Vision evaluations use eight domain-specific CLIP vision-encoder checkpoints across SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD, considering three ViT sizes.Source: §C.1 · p. 17
  • The language setup uses downstream models for eight GLUE datasets, finetuned from pretrained RoBERTa-Base and RoBERTa-Large.Source: §C.1 · p. 17
  • Autoregressive evaluations use WizardMath-13B and LLaMA-2-13B-Code-Alpaca, assessed on GSM8K, MATH, HumanEval, and MBPP.Source: §C.1 · p. 17

C.2 DETAILS OF BASELINE METHODS

The appendix describes the baseline implementations and the construction and adaptation settings for FDAs across vision, language, and autoregressive models. It also identifies the scope of several reported performance tables without providing their cell values here.

C.2 DETAILS OF BASELINE METHODS

  • Data-based baselines follow released implementations, while data-free baselines use public open-source code; the authors state they sought fair and strong baselines.ViT baseline results are taken from Xu et al. (2025), and the LLM task-arithmetic coefficient is set to 0.5 following that work.Source: §C.2 · p. 17

C.3 DETAILS OF FDAS

  • FDA experiments use layer-wise construction, consistent settings across layers, and both Gaussian and parameter initializations, with Gaussian σ set to 0.01.Source: §C.3 · p. 17
  • Vision and RoBERTa FDAs use 64 anchors per task, while autoregressive FDAs use 8192 anchors; construction uses AdamW with model-specific token counts and optimization steps.The vision and RoBERTa configurations use 1200 steps; the autoregressive configurations use 200 steps, all at learning rate 1e−2.Source: §C.3 · p. 17
  • Parameter adaptation uses Adam with batch size and training duration set by model family and initialization, including different equations for pretrained-model and task-vector-based initial models.The passage gives separate learning rates and epoch schedules for ViT, RoBERTa, WUDI-initialized ViT, and autoregressive models.Source: §C.3 · pp. 17–18
  • Tables 5–7 report merging performance for ViT-B/32, ViT-L/14, and RoBERTa-Base, respectively.The captions specify the model/task scope but provide no performance values in these passages.Source: Table 5 · p. 18, Table 6 · p. 18, Table 7 · p. 18

C.4 REMAINING RESULTS.

The remaining results report slight gains for WUDI-initialized merged models and give a practical heuristic for choosing the FDA scaling coefficient σ.

C.4 REMAINING RESULTS.

  • FDAs bring slight improvements to WUDI-initialized merged models, which already achieve 98.3% of the performance of eight individual models.The authors describe this initialization as already situated in a well-optimized local minimum.Source: §C.4 · p. 19

C.5 A PRACTICAL METHOD FOR CHOOSING THE SCALING COEFFICIENT

  • Choose σ by testing candidate values on one layer with the same initial FDAs, then select the value with the best alignment after a fixed number of iterations.The procedure is presented as a practical heuristic because σ is crucial for convergence.Source: §C.5 · p. 19

D MORE RESULTS AND DISUCSSIONS FOR SECTION 3

The additional analyses examine FDA spectra, alignment with real-data subspaces, feature-manifold proximity, and projection energy. They report long-tailed spectra and subspace alignment, but no clear evidence of closer manifold proximity; FDAs nevertheless alleviate overall representation bias.

FDA spectral evolution

  • FDAs initialized by different schemes tend to evolve into long-tailed structures during construction.The singular values are normalized by the largest value; the figure contrasts scaled random initialization with linear-weight initialization.Source: Figure 11 · p. 20

Subspace similarity

  • FDAs from different initialization schemes tend to align the subspace spanned by real data.The comparison tracks subspace similarity during FDA construction.Source: Figure 12 · p. 21

FDA proximity to real-data features

Projection energy

  • Projection-energy optimization is nearly intractable even for one layer, so the analysis computes attention-block and MLP energies separately and reports their average.For the merged model, task vectors are obtained by fine-tuning for one epoch; pretrained-model task vectors come from publicly available fine-tuned checkpoints.Source: D MORE RESULTS AND DISUCSSIONS FOR… · p. 22

Projection energy and representation effects

E MORE RESULTS AND DISCUSSIONS FOR SECTION 5

The ablations vary FDA initialization, shape, distance function, and construction steps. For RoBERTa-Base, the average performance trend with more tokens differs from ViT-B/32, largely because of STSB.

Experimental Details

  • The ablations vary initialization scheme, FDA shape and adaptation steps, distance function, and construction-phase optimization steps while holding the other stated settings fixed.The experiments use ViT-B/32 settings for initialization comparisons and keep batch size and adaptation learning rates fixed for shape comparisons.Source: E MORE RESULTS AND DISCUSSIONS FOR… · p. 23

Further analysis on token num

  • The token-count analysis reports performance by dataset, with Table 8 presenting results both including and excluding STSB.The table marks STSB as “–” in the results that exclude it.Source: Table 8 · p. 23, E MORE RESULTS AND DISCUSSIONS FOR… · p. 23
  • For RoBERTa-Base, average performance decreases as token count rises from 5 to 20, but without STSB it consistently increases with larger token counts.The reported performance drop is mainly attributed to STSB; the authors hypothesize that FDA shapes closer to the real-data space perform better.Source: E MORE RESULTS AND DISCUSSIONS FOR… · p. 23, Table 8 · p. 23

Related papers

Loading 2510.21223v1…