Source-linked AI summary
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
Wenjun Wang, Yanggan Gu, Shuo Cai, Yuanyi Wang, Pengkai Wang, Jianmin Wu, Hongxia Yang
TL;DR
Post-merge quantization couples low-bit reconstruction error with deviations already present in merged models, making direct PTQ unreliable. E-PMQ uses source experts and merged-weight anchoring during calibration, consistently improving low-bit merged-model quality across vision and text settings, including 4-bit GPTQ under Task Arithmetic from 65.0% to 73.6%.
Problem
Post-Merge Quantization targets merged models rather than independently trained models, coupling quantization deviation with expert-relative merging deviation during low-bit reconstruction.
Method
E-PMQ uses source expert weights for expert-guided layer-wise calibration targets and merged-weight anchoring to stabilize calibration while preserving the merged model’s integrated behavior.
Results
Across vision and text settings, E-PMQ consistently improves low-bit merged models over GPTQ, including 4-bit Task Arithmetic performance from 65.0% to 73.6%.
Takeaways & Limitations
E-PMQ supports effective low-bit post-merge quantization while retaining a single merged model for inference.
Takeaways & Limitations
E-PMQ requires source experts during pre-deployment quantization, adding quantization-stage memory and compute compared with naive PMQ baselines.
Abstract
from arXiv · showhide
Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increasingly practical low-resource strategy for integrating multiple task- or domain-specialized experts into a single model without joint training or multi-model serving. Together, quantization and model merging enable an efficient low-resource deployment pipeline by integrating multiple experts into one low-bit model. We formulate this setting as Post-Merge Quantization (PMQ). We show that directly applying post-training quantization (PTQ) to a merged model is unreliable because two distinct deviations are coupled: the quantization deviation introduced by low-bit reconstruction and the expert-relative merging deviation inherited from model merging. To mitigate these deviations, we propose E-PMQ, an expert-guided PMQ framework that uses source expert weights to provide expert- guided output targets during layer-wise calibration, together with merged-weight anchoring to stabilize the calibration and preserve the integrated behavior of the merged model. On CLIP-ViT-B/32 eight-task merging, E-PMQ improves 4-bit GPTQ from 65.0% to 73.6% under Task Arithmetic and from 69.1% to 74.8% under TIES-Merging. On harder settings, E-PMQ improves GPTQ from 34.8% to 76.7% on 20-task CLIP-ViT-L/14 and from 78.26% to 83.34% on FLAN-T5- base GLUE. These results demonstrate that E-PMQ enables effective post-merge quantization and low-bit deployment.
1 Introduction
The paper formulates Post-Merge Quantization (PMQ) as low-bit deployment for merged models and identifies coupled quantization and expert-relative merging deviations as the failure mode of naive PMQ. E-PMQ addresses this with expert-guided calibration targets and merged-weight anchoring, yielding consistent gains across evaluated vision and text settings.
- Problem setting: PMQ quantizes a merged model after expert integration, combining low-resource model merging with low-bit inference deployment.Model merging avoids joint training and multi-model serving while combining capabilities from specialized experts.
- Results: 65.0% to 73.6%: E-PMQ improves 4-bit GPTQ on CLIP-ViT-B/32 eight-task merging under Task Arithmetic.Under TIES-Merging, the corresponding improvement is 69.1% to 74.8%.
- Results: 34.8% to 76.7%: E-PMQ improves GPTQ on 20-task CLIP-ViT-L/14 under Task Arithmetic, while FLAN-T5-base GLUE improves from 78.26% to 83.34%.The paper reports consistent gains across merging methods, task scales, modalities, and quantization bit-widths.
- Failure mode: Naive PMQ couples low-bit reconstruction deviation with expert-relative merging deviation inherited from parameter composition.Directly reconstructing the merged model can therefore target a model already deviated from expert behaviors.
- E-PMQ: E-PMQ uses source expert weights for expert-guided output targets during layer-wise calibration and anchors quantized weights to the merged checkpoint.The guidance stabilizes calibration while merged-weight anchoring preserves the merged model’s integrated behavior.
2 Related Work
Related work covers model merging as a way to compose specialized models without joint training and post-training quantization as a low-bit compression method. In contrast, this work focuses on reliably quantizing an already merged model.
- Model Merging: Model merging composes multiple specialized models without joint training or deploying one model per task, using methods such as weight averaging, task arithmetic, and TIES-Merging.Other approaches include Fisher merging, DARE, and adaptive or data-free task-vector methods.
- Model Merging: Recent work frames model fusion as a scalable alternative to repeatedly training or serving many experts, while other methods use preference- or distillation-based composition.These studies emphasize building, scaling, or managing merged models rather than quantizing them.
- Post-training Quantization: This work instead studies how to quantize an already merged model more reliably than conventional PTQ assumptions support.The distinction arises because prior model-merging work centers on constructing or managing merged models, whereas PTQ generally preserves an independently trained full-precision target.
- Post-training Quantization: Post-training quantization compresses trained full-precision models into low-bit weights without end-to-end retraining through calibration-based rounding, scaling, or layer-wise reconstruction.Ordinary PTQ typically treats the full-precision model as the target to preserve, an assumption that is natural for independently trained models.
3 Preliminaries and Problem Formulation
This section defines PTQ, model merging, and post-merge quantization (PMQ), then shows why naive PMQ must account for both quantization and expert-relative merging deviations. The quantized model should preserve merged behavior without compounding deviations inherited from source experts.
- Post-Training Quantization: PTQ compresses a full-precision model into a low-bit model using a small calibration set without end-to-end retraining.Layer-wise reconstruction methods such as GPTQ formulate PTQ as minimizing reconstruction objectives.
- Model Merging: Model merging combines task- or domain-specialized experts into one model without joint training or per-task model deployment.The merged model can exhibit intermediate-representation discrepancies relative to its source experts.
- Post-Merge Quantization: PMQ obtains a low-bit model after merging by applying quantization to the merged model using a calibration set.A straightforward baseline directly applies standard PTQ to the merged model.
- Post-Merge Quantization: Naive PMQ is problematic because it treats the full-precision merged model as a reliable standalone reconstruction target despite pre-existing expert-relative deviations.These deviations are already present before quantization is applied.
- Post-Merge Quantization: PMQ combines low-bit quantization deviation with merging deviation from source experts, so quantization can compound errors and degrade propagated intermediate representations.The first deviation is introduced by quantization, while the second is inherited from model merging and remains invisible to objectives reconstructing only the merged model.
4 Method
E-PMQ performs forward-order, layer-wise post-merge quantization using calibration activations from the current partially quantized trajectory. It combines expert-guided output reconstruction with merged-weight anchoring and solves the resulting discrete objective using GPTQ-style sequential rounding.
- Layer-wise calibration: E-PMQ quantizes layers in forward order, collecting calibration activations from the current quantization trajectory for each expert.Earlier layers are already quantized or fixed, so activations reflect the partially quantized merged model’s encountered distribution.
- Expert-guided reconstruction: The layer-wise objective reconstructs expert-guided output targets instead of using only the full-precision merged output.Source expert weights provide the targets, while inputs come from the activation distribution the quantized merged model will actually encounter.
- Merged-weight anchoring: Merged-weight anchoring keeps the quantized weight close to the full-precision merged weight, preventing drift toward isolated experts.This stabilizes reconstruction when experts contain partially conflicting task-specific updates and helps preserve the merged model’s integrated behavior.
- Activation-adaptive anchoring: The anchor strength λℓ trades off expert-guided targets against preservation of the merged model and is adapted to each layer’s activation scale.The adaptive anchor scales with total calibration activation energy and adds diagonal loading to the layer’s quadratic form.
- Discrete optimization: Because the objective is constrained to the discrete low-bit space Qb, E-PMQ uses a GPTQ-style sequential rounding solver.The solver retains GPTQ’s implementation structure while optimizing the expert-guided objective with merged-weight anchoring.
5 Experiments
Across vision and language-model merging benchmarks, E-PMQ consistently improves 4-bit post-merge quantization over direct PTQ baselines, with larger gains in difficult multi-task and low-bit settings. Ablations show that merged-weight anchoring is essential, while expert guidance improves calibration efficiency and final accuracy.
- CLIP experiments: E-PMQ improves quantized CLIP-ViT-B/32 average accuracy over GPTQ by 11.4 points for Simple Averaging, 8.6 for Task Arithmetic, and 5.7 for TIES-Merging.Naive PMQ often loses accuracy after 4-bit quantization, especially when the upstream merger is weak.
- Scaling to more tasks: More than 27 points and 19.5 points are E-PMQ’s Task Arithmetic gains over full-precision merged models in 20-task CLIP-ViT-B/32 and CLIP-ViT-L/14, respectively.Increasing the number of merged tasks makes direct reconstruction less reliable because expert interference increases.
- Language-model experiments: 83.34 and 83.48 are E-PMQ’s average GLUE scores under Task Arithmetic and TIES-Merging, improving full-precision baselines of 78.79 and 79.98, respectively.E-PMQ outperforms RTN, GPTQ, and AWQ across both merging methods on FLAN-T5.
- Language-model experiments: 60.27 and 62.91 are E-PMQ’s average scores on Llama-3.1-3B and Llama-3.1-8B, exceeding GPTQ’s 58.71 and 61.66, respectively.E-PMQ achieves the best average performance among quantized variants at both model scales.
- Anchoring ablation: Removing merged-weight anchoring drops average accuracy from 68.00 to 5.37 under Task Arithmetic and from 72.20 to 4.57 under TIES-Merging.Positive anchor strengths remain stable across a reasonable range and consistently outperform direct GPTQ.
- Calibration efficiency: 72.23 average accuracy with 64 samples per task exceeds GPTQ with 256 samples per task by 7.20 points, while post-quantization inference remains single-model with identical parameter count and bit-width.E-PMQ uses 4-bit weight-only quantization by default and trades additional pre-deployment computation for more informative calibration.
6 Conclusion
Post-Merge Quantization (PMQ) is a distinct low-bit deployment setting because merged models may already deviate from source experts, making ordinary PTQ unreliable. E-PMQ addresses this through expert-guided calibration targets and merged-weight anchoring, yielding consistent gains over naive PMQ baselines across evaluated settings.
- 6 Conclusion: PMQ combines pre-existing expert-relative merging deviation with low-bit quantization deviation, so direct PTQ can preserve an imperfect reconstruction target.This distinguishes PMQ from ordinary post-training quantization of an otherwise direct model.
- 6 Conclusion: E-PMQ constructs calibration targets from source expert weights and uses merged-weight anchoring to stabilize low-bit calibration.The framework is designed to guide calibration toward expert-informed targets while anchoring it to the merged weights.
- 6 Conclusion: E-PMQ consistently outperforms naive PMQ baselines such as GPTQ across merging methods, task scales, and bit-widths.Experiments cover CLIP-based vision merging and FLAN-T5 language merging.
A Limitations … D.1 Full 8-Task Results on CLIP-ViT-L/14
E-PMQ improves 4-bit post-merge quantization across several CLIP merging settings, but requires source experts and additional calibration-time resources, with performance depending on calibration coverage. The method uses expert-guided targets and merged-weight anchoring in a GPTQ-style layer-wise solver, producing a single low-bit merged model for inference.
- A Limitations: E-PMQ requires source expert weights during pre-deployment quantization to construct expert-guided targets; otherwise, it reduces to direct post-merge quantization.Using expert weights also adds quantization-stage memory and compute compared with naive PMQ baselines.
- A Limitations: E-PMQ’s cost scales with the number of experts K and the calibration budget, making it best suited to compression settings where additional pre-deployment computation is acceptable.This extra cost is not incurred at inference time.
- A Limitations: After quantization, E-PMQ produces one low-bit merged model that requires neither experts, calibration data, nor additional inference-time modules.The experts and calibration resources are used during compression rather than deployment.
- A Limitations: E-PMQ depends on calibration samples that represent the experts’ task distributions, and its experiments focus on CLIP and FLAN-T5.Extending the method to larger-scale LLMs, diverse modalities, and more complex merging scenarios remains future work.
- B Algorithm: Algorithm 1 initializes from the merged weights, processes layers and experts sequentially, and collects expert/task activations for calibration.The procedure is organized layer by layer across all experts or tasks.
- B Algorithm: For each layer, E-PMQ uses a GPTQ-style discrete rounding solver under its objective, replaces the layer with the quantized result, and returns the quantized merged model.The deployed model is obtained through sequential low-bit layer replacement.
- C Continuous Relaxation and Solver Statistics: The continuous relaxation recasts E-PMQ’s layer-wise objective as a quadratic problem whose statistics incorporate expert-guided output targets and merged-weight anchoring.These statistics clarify the modifications made to the GPTQ-style solver.
- D Additional CLIP Results: The additional CLIP results cover a larger backbone and extended 14-task and 20-task suites containing datasets such as Flowers102, PCAM, FER2013, and Food101.The extended suites also include multiple image, character, and rendered-text classification datasets.
D.2 Full 14-Task Results · D.3 Full 20-Task Results
The full 14-task results show that E-PMQ consistently improves post-merge quantization across CLIP backbones, while the full 20-task results demonstrate larger gains and scalability to harder task suites.
- D.2 Full 14-Task Results: E-PMQ substantially improves over RTN, GPTQ, and AWQ on full 14-task CLIP-ViT-B/32 results, particularly for weaker upstream mergers such as Task Arithmetic.
- D.2 Full 14-Task Results: The full 14-task CLIP-ViT-L/14 results retain consistent gains, indicating that E-PMQ’s improvement extends beyond CLIP-ViT-B/32.
- D.2 Full 14-Task Results: The reported 4-bit PMQ evaluations measure top-1 accuracy (%) and compare changes against corresponding full-precision merged checkpoints.
- D.3 Full 20-Task Results: On full 20-task CLIP-ViT-B/32 results, naive PMQ becomes more challenging, whereas E-PMQ provides large improvements using expert-guided targets during quantization.
- D.3 Full 20-Task Results: E-PMQ again outperforms naive PMQ baselines across the main merging methods on full 20-task CLIP-ViT-L/14 results.
- D.3 Full 20-Task Results: The full 20-task CLIP-ViT-L/14 findings confirm E-PMQ’s scalability to larger task suites.
E LLM implementation details.
The Llama experiments merge four capability-oriented experts with Task Arithmetic, then evaluate 4-bit weight-only quantization on Llama-3.1 models using standardized calibration settings. E-PMQ uses source experts only during calibration and deploys a single merged model without inference-time expert modules.
- Llama model construction: Task Arithmetic merges four capability-oriented experts with merge coefficient 0.3 for Llama-3.1-3B and Llama-3.1-8B experiments.The experts cover instruction following, coding, mathematics, and multilingual understanding.
- Quantization and calibration: All quantized LLMs use 4-bit weight-only quantization with group size 32, calibration batch size 4, and maximum sequence length 512.Each method uses 256 calibration samples, and non-quantized tensors are stored in bfloat16.
- Baselines: The comparison includes E-PMQ, AWQ, GPTQ, and the full-precision merged model without quantization.AWQ uses grid-search parameter 20 in the reported baseline experiments.
- E-PMQ implementation: E-PMQ sets the global anchor scaling hyperparameter to α = 1 and uses source experts only during calibration to construct expert-guided targets.Expert model paths are instantiated from the corresponding model family and task name.
- Deployment: After quantization, deployment uses a single 4-bit merged model without additional experts or inference-time modules.This deployment configuration removes source experts from inference.
F Full Anchor-Strength Ablation
The full anchor-strength ablation shows that merged-weight anchoring is essential for stable E-PMQ performance. Removing the anchor causes severe collapse, whereas positive anchor strengths consistently outperform GPTQ in average accuracy.
- Anchor-strength ablation: Setting α = 0 removes merged-weight anchoring and causes severe performance collapse.The ablation evaluates anchor strength in the 4-bit CLIP-ViT-B/32 8-task PMQ setting.
- Anchor-strength ablation: Positive anchor strengths remain stable across tasks and consistently outperform GPTQ in average accuracy.The results are reported as top-1 accuracy (%) with average changes relative to full-precision merged checkpoints.
- Anchor-strength ablation: The ablation supports merged-weight anchoring as a necessary component of the E-PMQ objective.This conclusion follows from the contrast between collapse without anchoring and stable performance with positive anchor strengths.
G Implementation Details
The experiments use a fixed weight-only quantization protocol with task-scaled calibration data, while varying α, bit-width, or calibration budget in controlled analyses. Test sets are reserved exclusively for final evaluation.
- Quantization protocol: All quantized models use weight-only quantization with group size 128, calibration batch size 32, and evaluation batch size 128.The main experiments use 256 calibration samples per task.
- Quantization protocol: For a K-task merged model, the main calibration protocol uses 256K total calibration samples.This follows from using 256 calibration samples per task.
- Quantization protocol: α is set to 0.01 for Simple Averaging, Task Arithmetic, and TIES-Merging, versus 10 for WUDI-Merging.These settings are used in the implementation protocol.
- Controlled analyses: Bit-width analysis varies only the weight bit-width, whereas calibration-budget analysis varies calibration samples per task with other quantization settings fixed.Test sets are used only for final evaluation.