PaperlayerView exact revision on arXiv

Source-linked AI summary

FeatCal: Feature Calibration for Post-Merging Models

Yanggan Gu, Shuo Cai, Zihao Wang, Wenjun Wang, Yuanyi Wang, Pengkai Wang, Sirui Huang, Su Lu, Jianmin Wu, Hongxia Yang

arXiv:2605.13030v1cs.LGcs.AI

TL;DR

Merged models often underperform the task experts they combine, motivating a closer look at feature drift. FeatCal analyzes and reduces this drift through forward-order, layer-wise closed-form calibration, improving performance over Surgery and ProbSurgery across CLIP and GLUE benchmarks.

  • Problem

    Merged models often underperform the task experts they combine, leaving a persistent post-merging performance gap.

  • Method

    FeatCal analyzes feature drift and calibrates merged weights layer by layer in forward order using closed-form updates tied to merged weights.

  • Results

    FeatCal improves merged-model performance over Surgery and ProbSurgery across CLIP and GLUE, reaching 85.5% versus 77.0%/78.8% on CLIP-ViT-B/32 TA.

  • Takeaways & Limitations

    FeatCal reduces feature drift while preserving model-merging benefits without adding inference modules.

  • Takeaways & Limitations

    The analysis assumes task experts are fine-tuned from a common pretrained base and share the same architecture.

Abstract

from arXiv · show

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

1 Introduction

The paper explains post-merging underperformance through feature drift and develops FEATCAL to reduce it by calibrating merged weights layer by layer in forward order. FEATCAL uses closed-form updates without gradient descent or inference-time extra modules, improving accuracy, sample efficiency, and calibration cost.

  • Motivation: Model merging avoids joint training, retraining, and separate deployment, but merged models often underperform the task experts they combine.This performance gap motivates analyzing and correcting feature drift after merging.
  • Feature-drift analysis: Feature drift is decomposed into upstream propagation and local mismatch, which propagate through later layers and contribute to output drift.The analysis follows drift in forward order from expert input features to final outputs.
  • FEATCAL: FEATCAL calibrates merged weights layer by layer in forward order, keeps them close to the merged model, and solves each update in closed form.The regularization helps preserve model-merging benefits and reduce overfitting on small calibration sets.
  • Results: 85.5% versus 77.0%/78.8%: FEATCAL outperforms Surgery/ProbSurgery on CLIP-ViT-B/32 8-task TA; 85.2% versus 83.7%/82.2% on FLAN-T5-base GLUE.FEATCAL also improves TA by +2.0/+2.3 average points on MergeBench 3B/8B models.
  • Efficiency: 82.9% is reached with 8 examples per task, while 256 examples per task require 53 seconds, about 4x faster than both baselines.These results indicate lower calibration cost and strong sample efficiency under the same calibration protocol.

2 Related Work

Prior model-merging methods construct fused models through parameter-space merging or feature-informed layer updates, whereas FEATCAL starts from the merged model for post-merging calibration. Related post-merging methods use expert-guided feature interventions, while FEATCAL differs in parameterization and deployment by avoiding auxiliary intervention modules.

  • Model Merging: Model-merging methods build fused models directly in parameter space or derive layer updates from feature statistics, regression objectives, or explicit feature drift objectives.These approaches define how to build the merged model.
  • Post-Merging Feature Calibration: FEATCAL treats the merged model as its starting point and differs from related methods in parameterization and deployment by avoiding auxiliary intervention modules.Its calibration is folded into the model rather than learned or deployed through auxiliary intervention modules.
  • Post-Merging Feature Calibration: Post-merging alternatives calibrate merged-model features using task-specific plugins, deeper interventions, probabilistic feature-drift modeling, or parameter-efficient modules.These methods establish that expert-guided feature calibration is useful.

3 Post-Merging Feature Drift: Problem Formulation and Properties

The section formalizes layer-wise feature drift between merged models and task experts, decomposes it into local mismatch and propagated upstream drift, and connects final feature drift to output drift.

  • 3.1 Layer-Wise Feature Drift: The framework considers task experts and a merged model sharing an architecture and pretrained base, with layer functions evaluated over each task’s data distribution.For task i and layer ℓ, expert and merged features are defined recursively for the same input sample.
  • 3.1 Layer-Wise Feature Drift: Layer-wise feature drift is the difference between the merged-model feature and the corresponding task-expert feature at the same layer.This pointwise drift is the signal propagated through the subsequent analysis.
  • 3.2 Local Mismatch and Drift Propagation: Proposition 1 decomposes layer-wise drift into local mismatch at the current expert input and propagation of drift inherited from earlier layers.Local mismatch reflects differing merged and expert layer maps, while propagation reflects changed layer inputs caused by upstream drift.
  • 3.2 Local Mismatch and Drift Propagation: The layer-wise recursion is e_i,ℓ(x) = A_i,ℓ(x)e_i,ℓ−1(x) + m_i,ℓ(x), combining propagated drift with the current local mismatch.The recursion starts from e_i,0(x) = 0.
  • 3.2 Local Mismatch and Drift Propagation: Final feature drift is a downstream combination of local mismatches from different layers through compatible local sensitivity maps applied in forward order.For residual networks, skip connections carry upstream feature drift, and drift can grow under specific conditions.
  • 3.3 From Feature Drift to Output Drift: Output drift is defined through the difference between merged and expert task score vectors produced from their final features.The score settings include class logits, CLIP candidate scores, and fixed-prefix decoder vocabulary logits.
  • 3.3 From Feature Drift to Output Drift: Under the stated Lipschitz condition, final feature drift bounds a perturbation term in output drift, while score-map mismatch contributes an additional term.When the task score map is shared, the score-map mismatch term is 0.
  • 3.3 From Feature Drift to Output Drift: Feature drift can affect outputs and task loss; in language models, output-logit drift can change token probabilities and potentially alter the next-token choice.Softmax maps logits to token probabilities, and cross-entropy loss depends on the assigned probability.

4 FEATCAL: Feature Calibration for Post-Merging Models

FeatCal calibrates an already merged model layer by layer in forward order, using current calibrated-model features and expert-guided objectives with explicit regularization. Its linear-module updates are closed-form regularized regressions that preserve the architecture while reducing feature drift.

  • Forward-order calibration: FeatCal calibrates the merged model layer by layer in forward order, recollecting features after earlier updates before fitting each subsequent layer.This aligns calibration with the layer-by-layer propagation of feature drift.
  • Linear-module calibration: Linear modules receive closed-form regularized regression updates that replace merged weights without gradient descent, adapters, or inference-time modules.The procedure preserves the existing architecture; bias and LayerNorm affine updates are also designed but provide limited gains in practice.
  • Calibration objective: The basic per-module objective minimizes linear module feature-drift error while penalizing movement away from merged weights.The quadratic penalty makes this a tractable local surrogate rather than an exact objective for end-to-end task risk or all-layer feature drift.
  • Target interpolation: FeatCal interpolates the target feature to retain expert signal while reducing aggressive fitting of upstream drift already present in calibrated-model inputs.Using raw expert input features directly could force the current module to compensate for drift left by earlier layers.
  • Merged-base anchoring: The regularization anchor combines merged and base weights as W anc = ρW mer + (1 −ρ)W base, while λ > 0 controls regularization strength.The base model supplies a pretrained reference that may contain knowledge absent from a small calibration set.

5 Experiments

Experiments evaluate FeatCal across CLIP, FLAN-T5 GLUE, and MergeBench LLM merging, showing improvements over post-merging baselines alongside feature-drift diagnostics, sample-efficiency, cost, robustness, and ablation analyses.

  • Benchmarks: Experiments cover CLIP image classification, FLAN-T5 generation on 8 GLUE tasks, and Llama-family domain-expert merging across 6 MergeBench tasks.CLIP uses 8-, 14-, and 20-task suites; GLUE averages eight task scores, and MergeBench includes mathematics, coding, instruction following, and general knowledge.
  • Experimental setup: FeatCal is applied over multiple upstream mergers using 256 calibration samples per task, forward-order layer calibration, and closed-form linear-weight updates.Main CLIP runs also update bias and LayerNorm affine parameters, while FLAN-T5 calibrates linear-module biases but not LayerNorm affine parameters.
  • FLAN-T5 GLUE: +6.3 average points: FeatCal improves FLAN-T5-base Task Arithmetic, exceeds Surgery and ProbSurgery, and improves all 8 GLUE tasks.For FLAN-T5-large, gains are smaller but FeatCal still achieves the best post-Task Arithmetic average.
  • MergeBench LLM: +2.0 points and +2.3 points: FeatCal improves the 6-task MergeBench average over Task Arithmetic on the 3B and 8B models, respectively.It outperforms Surgery and ProbSurgery in both model blocks, with the largest gain being +15. points on IFEval for the 8B setting.
  • Feature-calibration diagnostics: 0.850 versus 0.785: FeatCal raises macro-average expert-feature cosine over TA with Surgery, while Stanford Cars shows a 0.084 mean per-sample gain.These diagnostics test whether performance gains coincide with final features closer to task experts.
  • Efficiency, robustness, and ablations: 82.9% with 8 examples per task and 53s at 256 examples: FeatCal reaches near-85.5% accuracy, runs 4.1× faster than Surgery, and remains strongest under corruptions.Across clean, Gaussian-noise, motion-blur, and fog conditions, FeatCal averages 78.0%, versus 71.5% for Surgery and 71.6% for ProbSurgery; ablations favor conservative calibration.

6 Conclusion

Feature drift explains how local mismatches propagate through merged models and affect outputs. FeatCal uses this signal for layer-wise closed-form calibration, improving CLIP and FLAN-T5 mergers over Surgery and ProbSurgery without adding inference modules.

  • Conclusion: Feature drift frames post-merging calibration by tracing how local mismatches propagate through merged models and affect outputs.This provides the conceptual basis for calibrating merged models.
  • Conclusion: FeatCal captures features layer by layer and applies closed-form updates to linear modules.The method uses the feature-drift signal directly for calibration.
  • Conclusion: FeatCal improves CLIP and FLAN-T5 mergers over Surgery and ProbSurgery in the main settings without adding inference modules.The approach preserves modular inference while improving merger performance.

A Limitations

FeatCal’s evaluation is limited to CLIP and FLAN-T5, requires development-set hyperparameter selection, and depends on task calibration data. Despite these constraints, experiments span more than 20 vision and language tasks, while calibration remains practical and data-efficient relative to Surgery and ProbSurgery.

  • Computational scope: More than 20 tasks across vision and language are covered, but experiments remain limited to CLIP and FLAN-T5 because of compute constraints.The study does not scale to additional modalities or much larger models.
  • Hyperparameter selection: λ, ρ, and α still require selection on a development or validation set, with no automatic selection rule.Fast closed-form calibration makes this search practical and adds little overhead compared with iterative calibration baselines.
  • Dependence on task calibration data: 8 examples per task already give strong gains on CLIP-ViT-B/32 TA, and performance saturates with fewer samples than Surgery and ProbSurgery.FeatCal nevertheless requires task calibration data to fit post-merging parameter updates, which may be limiting when task data cannot be stored or sampled.

B Additional Feature-Distribution Diagnostics for 8-Task TA … H Calibration Algorithm

The appendices validate FeatCal’s feature-drift perspective, connect layer-wise drift to residual propagation and output behavior, and formalize forward-order calibration with closed-form updates and an explicit algorithm. They also provide eight-task diagnostics and implementation details for biases and LayerNorm affine parameters.

  • B Additional Feature-Distribution Diagnostics for 8-Task TA: Figure 6 compares task-expert, TA, and TA w/ FEATCAL final projected features across all eight CLIP-ViT-B/32 TA tasks.Each task uses a separate joint 2D projection intended for within-task, not cross-task, comparison.
  • C Proofs for the Feature-Drift Analysis: The feature-drift proofs decompose each layer’s drift into propagated upstream error and local mismatch, with every final-drift term propagated through downstream sensitivity maps.The derivation iterates the layer recurrence from the first to the final layer.
  • D Residual Propagation and Conditional Growth: Residual connections preserve upstream drift through the identity skip, while the residual branch can compensate for or amplify it.Monotone amplification is conditional on local expansion along the current drift direction and insufficiently large opposing mismatch.
  • E Output-Drift Bridge: Output-score drift is bounded by final-feature drift under local Lipschitz conditions, while logit perturbations affect probabilities, losses, and decision margins.A decision changes only when relative score drift closes the corresponding expert margin.
  • F Forward-Order Calibration and Deployed Feature Sources: FeatCal collects current calibrated-model features before fitting each layer because later updates cannot alter that layer’s deployed prefix input distribution.Using merged-model, expert, or stale features can optimize an objective different from the one exposed by the final calibrated model.
  • G Bias and LayerNorm Affine Updates: The implementation calibrates biases after solving weights and updates LayerNorm scale and shift using current normalized features, with numerical stabilization for near-singular determinants.Unavailable merged or base anchors omit their corresponding regularization terms.
  • H Calibration Algorithm: Algorithm 1 processes layers in forward order, caches calibrated-model and expert features once per layer, and stores closed-form linear, bias, or LayerNorm updates.All stored parameters are loaded into the calibrated model after completing the layer’s module updates.

I Detailed CLIP Results · J Additional Diagnostic: Comparison with ProbSurgery

The detailed CLIP results report full top-1 accuracy tables across 8-, 14-, and 20-task settings for CLIP-ViT-B/32 and CLIP-ViT-L/14. Compared with TA w/ ProbSurgery, FEATCAL improves average feature alignment and accuracy, while cosine alignment does not fully explain task-level gains.

  • I Detailed CLIP Results: Tables 7–11 report full CLIP top-1 accuracy results across 8-, 14-, and 20-task settings for ViT-B/32 and ViT-L/14.Parenthesized Avg. values indicate changes over the corresponding upstream merger.
  • I Detailed CLIP Results: Fig. 7 compares feature-calibration diagnostics for TA w/ FEATCAL and TA w/ ProbSurgery on CLIP-ViT-B/32 Task Arithmetic.The panels examine final-feature cosine, per-sample cosine, cosine-versus-accuracy gains, and task-vector cosine.
  • I Detailed CLIP Results: Table 12 provides mean top-1 accuracy and standard deviation across three seeds for the 8-task Task Arithmetic sample-budget experiments.The entries correspond to Fig. 3.
  • J Additional Diagnostic: Comparison with ProbSurgery: 0.850 versus 0.814 macro-average expert cosine, 85.5 versus 79.1 8-task average accuracy, and +0.079 Stanford Cars mean expert cosine favor FEATCAL over ProbSurgery.These comparisons are reported in the repeated diagnostic.
  • J Additional Diagnostic: Comparison with ProbSurgery: FEATCAL improves accuracy over TA w/ ProbSurgery on every task, despite lower final-feature expert cosine on EuroSAT, GTSRB, and MNIST.This shows expert-feature cosine is useful but not a complete explanation of accuracy.
  • J Additional Diagnostic: Comparison with ProbSurgery: −0.074 is the largest negative cosine change on GTSRB, where accuracy still improves by +4.97 points.The diagnostic therefore separates feature-cosine changes from task-level accuracy changes.
  • J Additional Diagnostic: Comparison with ProbSurgery: Panel (d) shows that FEATCAL’s feature-space and parameter-space relationships retain the mismatch observed in Fig. 2.Although final-feature alignment improves on average, the supplied passage ends before specifying the remaining comparison.

K Efficiency Experiment Protocol

The efficiency study evaluates post-merging calibration on an 8-task CLIP-ViT-B/32 Task Arithmetic setting across controlled calibration budgets and seeds. It separates calibration resource measurement from final evaluation and uses shared hardware and software conditions.

  • Task and model setting: The study uses FusionBench CLIP-ViT-B/32 on eight image-classification tasks, with Task Arithmetic scaling factor 0.3 and all methods initialized from the same saved TA model.Surgery, ProbSurgery, and FEATCAL follow their native FusionBench calibration interfaces.
  • Data budgets and seeds: 7 calibration budgets—8, 16, 32, 64, 128, 256, and 512 examples per task—are evaluated, with mean and standard deviation reported over 3 seeds.Each method uses exactly n examples per task; Surgery and ProbSurgery set max_samples_per_task=n, while FEATCAL sets num_regmean_examples=n.
  • Measurement protocol: Calibration efficiency is measured using calibration-only wall-clock time, GPU energy, and CPU RSS, while final accuracy comes from a separate evaluation stage.The profiler evaluates the TA baseline and calibrated models only after all calibration jobs finish.
  • Shared hardware and software: Each run uses one NVIDIA A800-SXM4-80GB GPU, an Intel Xeon Platinum 8358 node with 128 logical CPUs, Python 3.12.2, and PyTorch 2.10.0 + cu128.The paper uses the same single-GPU worker class across its experiments.

L Corrupted-Calibration Robustness Protocol

The corrupted-calibration analysis evaluates post-merging methods in an 8-task CLIP-ViT-B/32 Task Arithmetic setting, corrupting only calibration data while measuring final accuracy on clean test sets. It tests Gaussian noise, motion blur, and fog at severity 3, with uncalibrated Task Arithmetic reported only in the clean column.

  • Protocol: The protocol uses FusionBench CLIP-ViT-B/32 with 8-task Task Arithmetic, starts every method from the same merged model, and evaluates on clean test sets.Only data consumed by post-merging calibration are corrupted.
  • Protocol: 256image-per-task calibration data are corrupted for FEATCAL, while Surgery and ProbSurgery use each method’s native calibration stream.The passage specifies the FEATCAL calibration-set size and distinguishes the baseline methods’ native calibration data.
  • Corruptions: Severity-3 Gaussian noise, motion blur, or fog is applied one corruption type at a time, and calibrated rows report final models from the corresponding corrupted-data runs.Uncalibrated Task Arithmetic has no calibration stream and is therefore reported only in the clean column.

M Additional MergeBench Expert Results · N Broader Context and Future Directions

The paper supplements its compact main-text results with detailed MergeBench expert references and situates FeatCal within broader model-fusion directions, including scaling and evaluation beyond accuracy.

  • M Additional MergeBench Expert Results: MergeBench Tab. 14 reports individual expert references omitted from the compact main-text table.These results provide the additional expert-level comparisons for MergeBench.
  • M Additional MergeBench Expert Results: All Tab. 14 entries are percentages.
  • M Additional MergeBench Expert Results: The reported average is the mean across MATH-500, GSM8K, HumanEval+, MBPP+, IFEval, and ARC-Challenge.
  • M Additional MergeBench Expert Results: HumanEval+ and MBPP+ use pass@1 as their reporting metric.
  • N Broader Context and Future Directions: FeatCal studies post-merging calibration after a base merger has produced one deployable model, within a broader model-fusion agenda.The paper frames model fusion as supporting scalable, sustainable, and more broadly accessible AI systems.
  • N Broader Context and Future Directions: A next step is testing how closed-form calibration scales with model size, task count, and available compute.
  • N Broader Context and Future Directions: Future calibration could incorporate preference signals and logits-level distillation rather than relying only on task labels or expert features.The paper identifies preference-oriented fusion, distillation, and logits-level transfer as complementary directions.
  • N Broader Context and Future Directions: Evaluation of calibrated merged models in interactive and multimodal settings should account for response uncertainty.
Loading 2605.13030v1…