PaperlayerView exact revision on arXiv

Source-linked AI summary

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Yihang Zhang, Shengke Sun, Junjie Wen, Feng Zeng

arXiv:2608.25354v1cs.LGcs.AIcs.CL

TL;DR

The paper addresses task interference in model merging caused by superposed, entangled task features. It projects task vectors into a high-dimensional sparse feature space with SAEs and uses GR-ZOO to select critical layers for disentanglement. Across Qwen2.5 evaluations, the method outperforms representative training-free merging baselines, including a 2.78% gain over the strongest baseline in a highly conflicting four-task setting.

  • Problem

    Task interference from superposition entangles task-specific features in parameter space, limiting conventional decomposition methods’ ability to isolate useful task directions.

  • Method

    The framework uses improved SAEs for high-dimensional sparse feature disentanglement and GR-ZOO to identify task-critical layers for selective merging.

  • Results

    The method consistently outperforms representative training-free merging baselines across mathematical reasoning, code generation, instruction following, and general knowledge tasks.

  • Takeaways & Limitations

    High-dimensional sparse disentanglement provides a model-merging approach that exhibits state-of-the-art performance and robustness in highly conflicting scenarios.

  • Takeaways & Limitations

    Training SAEs on critical layers adds computational overhead, and expansion-factor and cosine-similarity hyperparameters may require calibration across architectures or task scales.

Abstract

from arXiv · show

Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from \textit{superposition}, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78\% improvement over the strongest baseline.

1 Introduction

Model merging offers an efficient route to multi-task generalist models, but task interference arises when task vectors encode overlapping, entangled features. The proposed framework uses SAE-based sparse feature disentanglement and GR-ZOO layer selection to target these conflicts.

  • Model merging combines task-specific expert models into one generalist model without additional training.
  • Superposition causes task vectors to reuse and compete for overlapping feature directions, limiting direct arithmetic and orthogonal decomposition.
  • High-Dimensional Sparse Disentanglement Merging projects task vectors into a higher-dimensional sparse feature space before feature-level fusion.
  • SAEs map layer-wise task vectors into sparse feature codes so task-specific components can be separated more explicitly before being mapped back to parameter space.
  • GR-ZOO identifies task-critical layers using forward passes, avoiding the computational cost of applying SAE-based disentanglement to every layer.

2 Related Work

Model merging integrates task-specific experts without joint retraining, with prior work spanning parameter arithmetic and parameter decomposition. This paper extends sparse feature disentanglement to task-vector merging by projecting updates into a high-dimensional sparse latent space before fusion.

  • Model Merging: Model merging combines multiple task-specific expert models into one generalist model without joint retraining.
  • Parameter Arithmetic: Parameter arithmetic merges models directly in the original parameter space using interpolation or vector addition, with later methods adding sparsification and conflict resolution.
  • Parameter Decomposition: Parameter decomposition projects task updates into structured subspaces before fusion through alignment, compression, spectral truncation, rank selection, or subspace partitioning.
  • Motivation: Superposition stores many latent features in overcomplete, non-orthogonal directions, causing polysemantic parameter coordinates and making low-dimensional pruning or orthogonalization distort crucial capabilities.
  • Sparse Feature Disentanglement: The proposed framework applies Sparse Autoencoders to task vectors, projecting parameter updates into a high-dimensional sparse latent space for feature-level disentanglement before fusion.

3 Methodology

The framework merges task vectors through high-dimensional sparse representations, using improved SAEs for feature-level disentanglement and GR-ZOO to select task-critical layers efficiently.

  • 3.1 Problem Setup: Model merging constructs one model from multiple task-specific experts without original training data or additional full fine-tuning.
  • 3.1 Problem Setup: Task interference arises because superposed task vectors contain latent capability directions that remain entangled under coordinate-wise pruning or orthogonal decomposition.
  • 3.2 High-Dimensional Sparse Disentanglement via Improved SAEs: The method encodes layer-wise task vectors with an overcomplete SAE, fuses sparse latent features, and decodes the result back into parameter space.
  • 3.2 High-Dimensional Sparse Disentanglement via Improved SAEs: Top-K activation preserves salient latent magnitudes while limiting each input to a small number of active features, avoiding systematic shrinkage from L1 sparsity.
  • 3.2 High-Dimensional Sparse Disentanglement via Improved SAEs: Residual fitting trains inactive or rarely used latent units to explain reconstruction directions missed by active features.
  • 3.3 Group-Ranked Zeroth-Order Optimizer (GR-ZOO): GR-ZOO estimates layer sensitivity using forward evaluations across grouped parameters and random perturbation directions, avoiding explicit gradient computation.

4 Experiments

Experiments evaluate the framework on Qwen2.5 models across multi-task settings, layer-selection comparisons, and ablations. The method identifies critical layers with GR-ZOO and improves merging performance, especially as task conflicts increase.

  • Experimental setup: Experiments use Qwen2.5-7B for three-task merging and Qwen2.5-1.5B for high-conflict four-task analysis.The tasks include mathematical reasoning, code generation, instruction following, general knowledge, and safety alignment.
  • Critical layer identification: GR-ZOO stabilizes zeroth-order estimates and identifies task-critical layers using only forward-pass evaluations.Its selected layers achieve 62.31% recovery for Math versus a 65.73% full-gradient upper bound, and 41.46% for Code versus 46.34%.
  • Main multi-task merging results: 68.49 Average on Qwen2.5-7B surpasses Task Arithmetic’s 67.71 while retaining instruction-following and mathematical capabilities near losslessly.Conventional arithmetic methods drop IFEval performance to 38.45 and 36.78 in the reported comparison.
  • Ablation study: The complete SAE design raises the average from 39.30 to 40.18 in three tasks and from 35.24 to 36.48 in four tasks.Ablations compare projection spaces, layer-selection strategies, and individual SAE components.

5 Conclusion

The paper proposes High-Dimensional Sparse Disentanglement Merging to address parameter superposition in multi-task merging. SAE-based projection and GR-ZOO layer selection support strong performance and robustness under highly conflicting tasks.

  • Conclusion: The framework uses an improved Top-K Sparse Autoencoder to project overlapping updates into an orthogonal high-dimensional space.GR-ZOO identifies task-critical layers for targeted disentanglement.
  • Conclusion: Extensive evaluations report state-of-the-art performance and exceptional robustness in highly conflicting scenarios.

Limitations

The framework reduces task interference but adds computational overhead and introduces hyper-parameters requiring calibration across architectures or task scales.

  • Limitations: Training the Sparse Autoencoder on critical layers still adds computational overhead compared with instantaneous arithmetic methods such as Task Arithmetic.
  • Limitations: The high-dimensional expansion factor and cosine similarity threshold may require calibration for other architectures or substantially different task scales.The paper cites LLaMA-3 and Mistral as examples where optimal thresholds may vary.

Ethics Statement

The work uses publicly available datasets for model training and aggregate evaluation, without collecting new human-subject data or identifying individuals.

  • Ethics Statement: Evaluation relies exclusively on publicly available datasets and reports benchmark-level metrics rather than reproducing potentially harmful examples.
  • Ethics Statement: External datasets, benchmarks, models, and judges are attributed to their creators and used for their intended research purposes.

Supplementary Material

The supplementary material details expert construction and visualizes how the SAE-based merge differs from Task Arithmetic through sparse parameter calibration.

  • Training Details: Experts are initialized from the same foundation model and trained with full-parameter supervised fine-tuning.Training uses bfloat16 precision, DeepSpeed ZeRO-3, gradient checkpointing, and a cosine learning-rate schedule.
  • Expert Models: The Qwen2.5-7B experiments use separate Math, Code, and Instruction Following experts trained on automatically filtered datasets.The datasets contain approximately 7,160 mathematical reasoning samples, 8,646 programming tasks, and 5,473 instruction-following samples.
  • Parameter Visualization: The distributions are centered near zero and contain spike-like structures, indicating sparse and selective parameter calibration.The SAE-based merge largely preserves Task Arithmetic’s global structure while calibrating selected parameter directions.
  • Parameter Visualization: The compared weight difference is defined as ∆w = wOurs − wTA for selected parameter indices.wOurs and wTA denote the merged weights from the proposed method and Task Arithmetic, respectively.

C Computational Cost and Training Details

The computational-cost discussion describes GR-ZOO’s offline layer selection and the selective application of SAE processing without changing inference architecture.

  • Cost and Configuration: GR-ZOO and SAE costs are profiled separately, with SAE cost measured for one selected layer.The training configuration includes bfloat16 precision, DeepSpeed ZeRO-3, gradient checkpointing, and a cosine learning-rate schedule.
  • Selective Processing: GR-ZOO adds only an offline layer-selection cost before SAE processing is restricted to task-critical layers.This design avoids applying SAE-based disentanglement to every layer.
  • Inference: The decoded merged model preserves the original architecture and introduces no additional inference overhead.

D Disentanglement Analysis

Disentanglement analyses show that SAE representations reduce activation overlap, remain stable across hyperparameters, and allocate more features to individual tasks.

  • Activation Analysis: SAE reduces activation overlap and increases the proportion of task-specific features relative to raw task vectors.Overlap is measured with pairwise active overlap and active Jaccard similarity, while feature allocation separates task-specific from shared features.
  • Top-K Sensitivity: Performance remains stable across a broad range of Top-K sparsity levels rather than depending on a narrow setting.Individual tasks prefer slightly different K values, but overall variation is limited.
  • Threshold Sensitivity: The method is robust to different similarity thresholds, with the default τ = 10^-4 providing the best overall performance.These evaluations fix K = 32 while varying the similarity threshold.
  • Analysis Scope: The supplementary analyses position these results alongside detailed downstream, projection-space, layer-selection, and component ablations.

F.1 Detailed Downstream Results

Detailed ablations show that the complete SAE design and GR-ZOO provide the strongest overall balance across merged capabilities, while representation metrics expose trade-offs hidden by reconstruction error alone.

  • Three-Task Merging: 40.18 versus 39.30: the complete SAE design improves the three-task metric average over the standard L1 SAE.Gains occur on GSM8K, Humanities, and Others while HumanEval remains comparable.
  • Four-Task Merging: 36.48 versus 35.24: the complete SAE design improves the four-task metric average under stronger cross-task interference.General-domain scores increase for STEM, Social Sciences, Humanities, and Others.
  • Trade-offs: The proposed variant does not improve every metric: HumanEval falls from 32.93 to 32.32 and BeaverTail from 52.97 to 50.55.The authors interpret the gain as improved overall cross-task balance rather than uniform superiority.
  • Projection Space: The SAE achieves a 31.56 metric average, compared with 30.80 for PCA and 31.01 for random projection.The result indicates that the gain cannot be explained solely by using a different or higher-dimensional projection space.
  • Layer Selection: GR-ZOO achieves the highest metric average among the evaluated layer-selection strategies and better balances preservation across capabilities.Fisher selection performs better on GSM8K and IFEval but substantially underperforms GR-ZOO on HumanEval.
  • SAE Components: Removing Top-K lowers FVU but raises active-feature ratio from 8.33% to 35.76%, weakening sparse feature separation.Reconstruction error alone is therefore insufficient for evaluating latent-representation quality.
  • SAE Components: Removing orthogonality raises atom coherence from 0.0176 to 0.0224 and dead-feature rate from 0.0488% to 1.5299%.Removing both decoder normalization and orthogonality raises the dead-feature rate to 5.1921%.

H Detailed Proofs

The proofs formalize task-vector superposition under a behavior-aware metric, show why orthogonal decomposition cannot remove conflict without capability loss, and establish sparse SAE decomposition as an alternative. The framework then uses SAE fusion and GR-ZOO-selected critical layers for selective merging.

  • Lemma 1: Task-vector superposition arises when latent capability directions are mixed in weight space and are not behaviorally orthogonal.The observable decomposition follows from activation shifts mapped through the linearized operator, with approximation errors and behavior-invisible components.
  • Theorem 2: A sparse overcomplete decomposition with low dictionary coherence reduces cross-task capability conflict while approximately preserving the merged update.Theorem 2 contrasts sparse-code merging with raw task-vector merging through conflict and reconstruction-deviation bounds.
  • Theorem 2: SAE-based merging recovers sparse latent capability coordinates, merges task codes in the disentangled feature space, and maps the result back to parameter space.This approach is motivated as an alternative to merely orthogonalizing task vectors.
  • Theorem 1: Orthogonal decomposition cannot fully remove superposition-induced conflicts while preserving both interacting capability directions.Theorem 1 applies to projection or filtering operators such as PCA- or SVD-based merging.
  • Theorem 1: Capability conflict can disappear under orthogonal projection only by substantially distorting or discarding at least one capability direction.Preserving both directions leaves residual interference under the stated nonzero-interaction condition.
  • Algorithm 1: The workflow computes task vectors, ranks task-alignment gradients with GR-ZOO, selects the top-K critical layers, and performs high-dimensional SAE disentanglement and fusion.Algorithm 1 identifies selective merging as the computational workflow after critical-layer selection.
Loading 2608.25354v1…