Source-linked AI summary
CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging
Mingjie Zheng, Zihao Chen, Wenqing Chen, Weile Yuan, Zhixuan Chu, Jianxing Yu, Zibin Zheng
TL;DR
Model merging seeks to combine expert capabilities efficiently despite parameter interference and limited behavior-level feedback from naive merges. CoMerge uses self-supervised preference optimization with naive-merge defects as hard negatives to refine lightweight tensor-wise coefficients. It achieves 0.9968 average normalized performance on MergeBench and improves conflict-sensitive task results while optimizing 1,445 coefficients.
Problem
Model merging can suffer parameter interference, while existing approaches generally do not directly learn from degraded behaviors produced by naive merging.
Method
CoMerge constructs preference pairs from expert responses and naive-merging defects as rejected samples, then preference-optimizes lightweight tensor-wise merging coefficients without external annotations.
Results
0.9968 average normalized performance on MergeBench: CoMerge outperforms evaluated data-free and data-driven model-merging baselines while optimizing only 1,445 coefficients.
Takeaways & Limitations
CoMerge provides a conflict-driven explicit merging framework that preserves task-specific capabilities while reducing parameter-space interference and GPU-resource cost under the reported conditions.
Takeaways & Limitations
The expert-as-chosen and naive-merge-as-rejected assumption can introduce label noise because either response may be imperfect or only marginally different.
Abstract
from arXiv · showhide
Model merging provides an efficient paradigm for constructing multi-task large language models (LLMs) without full model retraining, yet it remains challenged by parameter interference. While existing methods aim to preserve the capabilities of individual expert models and mitigate interference, they generally do not directly learn from the potentially degraded behaviors exposed by naive merging. In this paper, we propose a conflict-driven preference optimization framework for model merging (CoMerge), which reformulates model merging as a preference optimization problem. The approach utilizes a self-supervised, conflict-driven strategy that leverages the defects of naive merging methods (e.g., task arithmetic) as hard negative samples to construct preference pairs without external annotations. By applying preference optimization to refine lightweight, tensor-wise merging coefficients, CoMerge enables the model to mitigate parameter-space conflicts while preserving task-specific capabilities. Extensive experiments show that CoMerge achieves an average normalized performance of 0.9968 on MergeBench, outperforming all evaluated data-free and data-driven model-merging baselines. Furthermore, on Llama-3.1-8B-Instruct, CoMerge yields marked improvements on conflict-sensitive tasks such as instruction following and safety, while remaining highly competitive with full-parameter fine-tuning despite optimizing only 1,445 scalar coefficients.
1 Introduction
Model merging combines expert capabilities efficiently but can suffer parameter interference and degraded behavior. CoMerge reformulates merging as self-supervised preference optimization, using naive-merge defects as hard negatives to refine lightweight coefficients.
- Model merging integrates task-specific experts in parameter space but conflicting task-vector directions can degrade performance under direct arithmetic aggregation.
- Existing methods mitigate interference through heuristics or data-driven optimization but generally lack behavior-level negative feedback from naive merged outputs.
- CoMerge constructs preference pairs from expert outputs and defective naive-merging outputs without external annotations.
- CoMerge applies preference optimization to lightweight, tensor-wise merging coefficients to mitigate parameter-space conflicts.
- 0.9968 on MergeBench: CoMerge outperforms evaluated model-merging baselines while optimizing only 1,445 coefficients and using 60.1% fewer GPU-minutes to reach peak performance than Full-DPO under the stated condition.The comparison uses the same preconstructed preference dataset for both methods.
2 Related Work
Related work spans data-free merging, data-driven calibration, and preference optimization. CoMerge differs by adding behavior-level negative feedback from naive merging to explicit model merging.
- Data-free Model Merging: Data-free merging combines fine-tuned models without retraining but addresses interference through operations such as sign resolution, sparsification, and update trimming.
- Data-driven Model Merging: Data-driven merging uses task data, model responses, or representations to learn coefficients, localized regions, or structured transformations for fusion.
- Preference Optimization: Preference optimization methods align models using preference data, while model-merging variants transfer capabilities through output or distribution alignment.
- Preference Optimization: Unlike implicit preference-optimization merging approaches, CoMerge applies preference optimization to explicit merging and avoids external reward-model ranking during preference-data construction.
3 Method
CoMerge formulates multi-task model merging as self-supervised preference optimization, using naive-merge failures as negative feedback while refining tensor-wise coefficients. Its pipeline extracts efficient task vectors, synthesizes preference pairs, and optimizes the merge against conflict-sensitive behaviors.
- Problem Formulation: Direct task-vector aggregation can degrade performance when different tasks induce conflicting updates at shared parameter coordinates.
- Conflict Mining with Preference Synthesis: CoMerge replaces a calibration objective with self-generated preference pairs contrasting expert outputs as chosen responses against Task Arithmetic outputs as rejected responses.The rejected outputs expose failure modes such as hallucinations, logical inconsistency, and domain or style mismatch.
- Framework Overview: The framework has three stages: task-vector extraction, self-supervised preference synthesis, and parameter-efficient preference optimization.It preprocesses task updates with global magnitude pruning and selective SVD-based low-rank approximation before optimizing the merge.
- Multi-Layer Efficient Task Vector Extraction: Task vectors retain the globally largest 10% of parameter updates by magnitude, followed by selective low-rank SVD when factorization reduces storage.Projection matrices are decomposed only when their low-rank factors require fewer elements than the pruned matrix; other tensors remain pruned and undecomposed.
- Parameter-Efficient Preference Optimization: CoMerge learns a separate coefficient λp,i for each task and mergeable parameter tensor instead of using Task Arithmetic’s shared scalar α.For Llama-3.1-8B-Instruct, this parameterization uses P = 289 tensors and N = 5 tasks, yielding 1,445 trainable scalar coefficients while keeping embeddings and the LM head fixed.
- Parameter-Efficient Preference Optimization: The coefficient-learning objective maximizes the likelihood margin between expert-positive and conflict-associated negative samples using DPO.The initial merged model is used as a frozen reference policy to constrain deviation from the initial merging manifold.
4 Experiments
Experiments evaluate CoMerge across tasks, model architectures, scales, negative-sample constructions, ablations, and computational costs. CoMerge achieves the strongest reported MergeBench average, generalizes across backbones, benefits from conflict-driven negatives and tensor-wise coefficients, and reaches comparable peak performance with substantially fewer resources than Full-DPO.
- Experimental Setup: Experiments use adapted MergeBench across 12 datasets in five categories, three backbones, five expert models per backbone, and data-free and data-driven baselines.The evaluated backbones are Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Gemma-2-2b-it.
- Main Results: 0.9968 average normalized performance on MergeBench is CoMerge’s highest reported merging-method score, exceeding Task Arithmetic by +0.0430 and AdaMerging (Gen.) by +0.0180.On Instruction Following and Safety, CoMerge scores 0.9952 and 1.0340, while Task Arithmetic reaches 0.8687 on instruction following.
- Scalability and Robustness Across Models: 0.9699 ± 0.0032 on Llama-3.2-3B-Instruct exceeds Dataless L&S at 0.9649 across all three seeds, while 0.9321 ± 0.0014 on Gemma-2-2b-it is comparable to RegMean++ at 0.9324.The results indicate stronger gains on Llama-3.2-3B-Instruct and competitive performance on Gemma-2-2b-it.
- Ablation Studies: Adding Task-Arithmetic-derived rejected responses raises average NP from 0.9842 to 0.9968 and Instruction from 0.9283 to 0.9952 compared with positive-only coefficient SFT.Replacing those negatives with base-model outputs instead reduces average NP to 0.9497 and Instruction to 0.8138.
- Ablation Studies: Replacing tensor-wise coefficients with global coefficients reduces performance from 0.9968 to 0.9822, while removing both negative feedback and tensor granularity lowers average performance to 0.9679.The joint ablation is below both the positive-only variant at 0.9842 and the global-coefficient variant at 0.9822.
- Visualization Analysis: Figure 3 shows task- and depth-dependent coefficient patterns: Instruction is larger in shallow-to-middle layers, whereas Safety, Coding, and Multilingual increase later.Math remains comparatively small with mild late-layer increases, and several middle layers exhibit lower averages.
- Negative Sample Construction: LLM-as-a-judge relabeling does not improve performance, while iterative negatives provide modest gains only when mixed with D1 and cannot replace it.The results support Task Arithmetic outputs as stronger endogenous negatives than generic response-quality relabeling in these experiments.
- Computational Efficiency and Hardware Accessibility: 1,445 coefficients on one H20 GPU reach comparable peak performance to Full-DPO using four H20 GPUs, reducing GPU-resource cost to peak by 60.1%.CoMerge uses 38.3 GPU-minutes versus 95.9 GPU-minutes for Full-DPO under the same preconstructed preference dataset.
5 Conclusion
CoMerge learns tensor-wise merging coefficients from expert responses and Task-Arithmetic-derived negatives through self-supervised preference optimization. Across the evaluated settings, it outperforms model-merging baselines while using a small number of coefficients and reduced GPU-resource cost.
- 0.9968 Avg. NP on the adapted MergeBench protocol was achieved by CoMerge on Llama-3.1-8B-Instruct.
- CoMerge outperformed all evaluated model-merging baselines and approached Full-DPO.
- 1,445 coefficients were optimized, while GPU-resource cost to peak fell by 60.1% under the same preconstructed preference dataset.
- Across three seeds, every CoMerge run exceeded the strongest reported merging-baseline score on Llama-3.2-3B-Instruct.
- CoMerge’s mean performance on Gemma-2-2b-it remained comparable to the best reported baseline.
- Task-Arithmetic-derived negatives produced substantially higher downstream performance than base-model negatives in a controlled Llama-3.1-8B-Instruct comparison.
Limitations
CoMerge’s self-supervised preference construction has limitations because expert and naive-merged responses do not always cleanly represent preferred and rejected behavior. Fixed preprocessing and scaling choices also remain insufficiently tested across backbones.
- Expert responses can be imperfect, while naive merged responses may remain acceptable or differ only marginally, introducing label noise.
- Generic score-based LLM-as-a-judge relabeling or filtering was unreliable for improving preference data.
- Pruning retention, maximum SVD rank, and Task Arithmetic scaling are fixed across backbones, leaving their sensitivity and individual contributions untested.
Ethical Considerations
The paper evaluates safety-related model merging under controlled conditions but notes that improved multi-task capability may still create misuse risks. Its safety-related data use is limited to existing datasets and offline experiments.
- Improved multi-task capability may introduce potential misuse risks if merged models are deployed without appropriate safeguards.
- Safety-domain preference data uses 1,000 prompts sampled from the existing synthetic WildJailbreak dataset under its license and responsible-use conditions.
- Safety benchmarks are used only for evaluation and analysis, while no new user data or additional harmful prompts are collected for preference construction.
A Implementation Details
Implementation uses tensor-wise coefficient optimization with fixed pruning, low-rank approximation, preference-generation, and evaluation settings. The experiments use distinct software environments for optimization and evaluation, with controlled comparisons and multi-seed testing.
- CoMerge uses effective batch size 16, learning rate 10^-2, DPO β = 0.1, λinit = 0.4, and 280 optimization steps.
- Preference generation samples with temperature 0.7, top-p 0.9, and at most 1,024 new tokens; the default training seed is 42.
- After global pruning, torch.svd_lowrank uses target rank 1500 only when factorization reduces storage; otherwise pruned matrices are retained.
- The optimization environment uses LLaMAFactory 0.9.2.dev0, Python 3.10.18, PyTorch 2.5.1+cu124, Transformers 4.45.2, and related training libraries.
- Safety evaluation uses a local safety-eval fork, while general-capability evaluation uses lm-evaluation-harness with locally added task configurations.
- Dataset scores are normalized by expert-model scores, with higher-is-better conversions for safety rates before macro-averaging.
- AdaMerging (Gen.) uses matched prompts, pruning, SVD rank, initialization, tensor-wise parameterization, batch size, and one-epoch optimization for controlled comparison.
- CoMerge is run with seeds 42, 43, and 44 on smaller backbones using the same pre-generated preference-pair pool and fixed settings.
B Additional Sweep for Iterative Preference Data
Table 7 presents a targeted sensitivity sweep of iterative preference-data mixtures, probing gain boundaries and whether retaining D1 is necessary.
- Table 7 evaluates iterative preference-data mixtures against D1.The sweep is framed as additional sensitivity analysis rather than an exhaustive search.
- The sweep expands along D3 for Llama-3.2-3B-Instruct and D2 for Gemma-2-2b-it.Each expansion follows the more promising direction identified for that model.
- The analysis probes gain boundaries and the necessity of retaining D1.
C Additional Results
Table 8 reports normalized MergeBench performance for Llama-3.2-3B-Instruct and Gemma-2-2b-it, including single-run comparisons and multi-seed CoMerge averages.
- Table 8 compares normalized MergeBench performance across Llama-3.2-3B-Instruct and Gemma-2-2b-it.
- The upper panel reports single-run per-category comparisons, with CoMerge corresponding to seed 42.
- The lower panel reports CoMerge Avg. NP across seeds 42, 43, and 44 as mean ± sample standard deviation.