PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging

Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Mengjie Zhao, Yunpu Ma, Kang Liu, Zihan Wang, Shi Feng, Daling Wang, Hinrich Schütze

arXiv:2609.37564v1cs.CL

TL;DR

Whole-vector model merging can overlook distinct geometric changes within task updates, allowing one component to affect another's aggregation. DiGA decomposes updates relative to pretrained weights and merges components separately; experiments report improved performance and reduced capability degradation across language and vision-language experts.

  • Problem

    Whole-vector merging overlooks heterogeneous geometry within task updates, and adaptive decisions based on complete-update statistics can couple how distinct components are aggregated.

    Source: §2.2 · pp. 2–3

  • Method

    DiGA uses pretrained weights as a shared geometric reference to decompose task updates into orthogonal components, merge corresponding components independently, and recombine them.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Experiments across language and vision-language experts show that DiGA improves merged-model performance and reduces capability degradation.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Component-wise processing is supported as a general, reusable merging principle across existing merging methods.

    Source: §1 · p. 2, §6 · p. 10

    Whole-vector aggregation does not explicitly distinguish geometric effects, and adaptive decisions from complete-update statistics can couple their treatment.

    Source: §2.2 · pp. 2–3

Abstract

from arXiv · show

Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on https://github.com/wzj1718/DiGA.

1 INTRODUCTION

Model merging consolidates specialized pretrained models without additional training, but whole-task-vector aggregation can couple distinct geometric changes. DiGA separates and independently merges those components, with experiments reporting improved performance across diverse settings.

1 INTRODUCTION

  • Model merging offers a way to consolidate fine-tuned domain experts into one model without additional training, avoiding the high cost of pretraining large models from scratch.The introduction frames this as an alternative for combining diverse capabilities.Source: §1 · p. 1
  • A task vector can encode both orientation changes to pretrained parameter structure and structural changes within or beyond it.Existing methods generally process the complete task vector as one merging unit.Source: §1 · pp. 1–2
  • Statistics computed from a complete task vector can let one geometric component alter how another is selected, weighted, or combined.The paper distinguishes this within-update interference from conflicts among different experts' task vectors.Source: §1 · p. 2
  • DiGA uses pretrained weights as a shared geometric reference, aggregates corresponding components in their respective subspaces, and recombines them into a unified update.This component-wise process is intended to prevent one component family's statistics from affecting another's aggregation while retaining the underlying merging operator's semantics.Source: §1 · p. 2
  • DiGA identifies cross-component coupling and orthogonally separates task updates into orientation and structural components for independent merging and recombination.The paper reports evaluations across diverse models, tasks, and merging methods, with consistent performance improvements and broad compatibility.Source: §1 · p. 2

2 BACKGROUND

Task-vector merging aggregates differences between task-specific experts and shared pretrained weights, while commonly treating each update as indivisible. Because updates can contain heterogeneous geometric changes, whole-vector and adaptive aggregation may couple how those changes are handled.

2.1 TASK-VECTOR MERGING

  • Task-vector merging aggregates expert updates relative to shared pretrained weights, and existing methods generally treat each update as one indivisible unit.The merging operator can differ across methods.Source: §2.1 · p. 2

2.2 LIMITATIONS OF WHOLE-VECTOR AGGREGATION

  • Fine-tuning can produce changes that reorient pretrained parameter structure or alter its internal geometry and extend beyond it.Whole-vector aggregation does not explicitly distinguish these effects.Source: §2.2 · p. 2
  • Adaptive methods derive decisions from complete-update statistics, allowing one type of geometric change to affect how another is aggregated.The paper calls this phenomenon cross-component coupling and poses disentangling and independently merging the components as the motivating questions for DiGA.Source: §2.2 · p. 3

3 METHODOLOGY

DiGA uses the pretrained weights as a shared geometric reference to separate task updates into orientation and structural components. It aggregates corresponding components independently, then recombines them into a unified update.

3.1 GEOMETRIC DISENTANGLEMENT OF TASK UPDATES

  • Separate TIES processing achieves a higher average at every tested nonzero structural strength λ, while whole-vector decisions increasingly drift from separately aggregated orientation components.The comparison holds magnitude masks fixed while varying the structural contribution.Source: Figure 1 · p. 4, §3.1 · p. 4
  • With the pretrained frame as reference, each update has an orthogonal decomposition into orientation changes and structural changes within or beyond that frame.The orientation component preserves the frame's structure; the structural component alters its internal geometry or extends beyond its span.Source: §3.1 · pp. 3–4

3.2 CROSS-COMPONENT COUPLING

  • Although components are orthogonal, statistics of complete updates can mix their information, letting one component influence adaptive merging decisions for another.TIES sign-election and expert selection use complete task updates, so changing a structural component can alter aggregation of a fixed orientation component.Source: §3.2 · pp. 4–5

3.3 COMPONENT-WISE AGGREGATION

  • DiGA applies a compatible aggregation operator independently to each component family and orthogonally recombines the resulting updates with the pretrained weight.Independent statistics prevent one component family from affecting the other's aggregation decisions; the framework is not tied to one merging rule.Source: §3.3 · pp. 5–6
  • For each component family, DiGA orthogonalizes unit directions while retaining source norms, then combines retained directions into a nonredundant aggregate that preserves source influence.Gram–Schmidt is applied to orientations rather than magnitudes.Source: §3.3 · p. 5
  • DiGA calibrates each aggregate direction's magnitude using mean source norm and directional consensus, retaining the mean scale at zero agreement and adjusting it with agreement.The affine rule is γ = (1 + c̄)n̄; positive agreement increases the scale, while negative agreement attenuates it.Source: §3.3 · pp. 5–6

4 EXPERIMENTS AND RESULTS

DiGA is evaluated across language-only and vision-language experts, with ablations and geometric analyses examining component separation. Across the reported settings, it exceeds the strongest compared baseline and its benefits extend to existing merging operators.

4.1 EXPERIMENTAL SETUP

  • The evaluation spans data-free merging methods, LoRA and OFT experts, fully fine-tuned language experts, and vision-language experts across target and applicable transfer benchmarks.DiGA is applied to attention and MLP projection matrices, while parameters outside the decomposition are merged using TA.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 59.18% (LoRA) and 58.60% (OFT) are DiGA’s target-task averages, beating the strongest baselines by 0.48 and 1.28 points, respectively.Transfer averages exceed the 37.82% base-model average in both settings, and DiGA achieves the highest ARC-Avg under both LoRA and OFT.Source: §4.2 · pp. 6–7
  • DiGA scores 41.09% versus the strongest baseline’s 40.87% on MergeBench, and 61.84%, 0.18 points above the strongest baseline, on vision-language experts.On the vision-language benchmarks, it achieves the highest PathVQA score, 67.61%.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • Across two to five OFT experts, DiGA achieves the largest gain over the base model at each count, with its largest advantage at five experts.The reported scaling trend is within this expert suite.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 ABLATION STUDIES

  • Task Avg. falls 6.70 points with T-only and 3.04 with R-only; removing source-magnitude weighting costs 3.42 points.Both components contribute complementary task information, with the structural component accounting for a larger share of retained performance.Source: Table 4 · p. 7, §4.3 · p. 8
  • Whole-vector aggregation lowers Task Avg. from 58.60 to 58.24 and Transfer Avg. from 40.92 to 40.86 compared with component separation.The whole-vector control uses DiGA’s aggregation rule but applies it to complete task updates rather than separating the component families.Source: §4.3 · p. 8

4.4 COMPONENT SEPARATION AS A GENERAL MERGING PRINCIPLE

  • For TIES, DARE, and TSVM, component separation improves task and transfer averages; in TSVM, separate processing beats joint processing by 2.62 Task Avg. points.For TSVM, Component-Separate improves Task Avg. by 0.37 points over TSVM, while Component-Joint decreases it by 2.25 points.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 GEOMETRIC EFFECTS OF MERGING

  • DiGA achieves the strongest OFT Task Avg. without the largest angular distortion; norm ratios stay near one while component separation changes update direction.The results suggest that effective merging involves controlled reorganization of directional structure, rather than simply preserving expert geometry as closely as possible.Source: §4.4 · pp. 8–9, §4.5 · p. 9, Figure 3 · p. 9

5 RELATED WORK

Prior work describes structured adaptation geometry and a range of model-merging approaches. This work instead separates geometrically distinct components within each task update before aggregation.

5 RELATED WORK

  • Downstream adaptation can involve structured, low-dimensional changes; OFT models rotations that preserve pairwise angles, while qGOFT permits controlled norm and angle changes.The paper asks whether directional-reorganization geometry can be isolated within general task updates and used for merging.Source: §5 · p. 9
  • Model merging spans weight averaging, task arithmetic, and methods using sparsification, sign resolution, parameter statistics, or adaptive weighting.TSVM uses singular-vector representations with orthogonalization, while Iso-CTS models common and task-specific subspaces.Source: §5 · p. 9
  • Unlike approaches that structure task representations, this work separates geometrically distinct components within each task update before aggregation.The paper frames its distinction relative to TSVM and Iso-CTS.Source: §5 · pp. 9–10

6 CONCLUSION

The paper identifies cross-component coupling as an overlooked source of interference and proposes DiGA to merge orthogonal task-update components separately. Experiments and applications to existing methods support improved performance and a reusable component-wise merging principle.

6 CONCLUSION

  • DiGA decomposes task updates into orthogonal orientation and structural components, merges them separately, and recombines them to address cross-component coupling.The paper identifies coupling as interference arising when heterogeneous components share aggregation decisions.Source: §6 · p. 10
  • Experiments across language and vision-language experts show that DiGA improves merged-model performance and reduces capability degradation.The conclusion reports these outcomes across both expert settings.Source: §6 · p. 10
  • Results with existing merging methods support component-wise processing as a general and reusable merging principle.The paper presents this as a conclusion from applying the approach to existing methods.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools to improve manuscript readability and language quality, with modifications reviewed by the authors.

AI USE STATEMENT

  • Generative AI tools were used to improve the manuscript's readability and language quality.The statement describes the tools' role as language editing.Source: AI USE STATEMENT · p. 10
  • The authors reviewed and verified all AI-assisted modifications.The statement assigns review and verification to the authors.Source: AI USE STATEMENT · p. 10
  • The authors take full responsibility for the final content, including text, claims, mathematical formulations, and experimental results.The responsibility statement explicitly covers these content categories.Source: AI USE STATEMENT · p. 10

A EXPERIMENTAL SETUP

The evaluation covers language-only and vision-language experts, with protocols assessing overall merging performance, the proposed geometric disentanglement, and its generality across operators. It includes both parameter-efficient and full fine-tuning settings.

A EXPERIMENTAL SETUP

  • The setup compares data-free merging methods under fixed coefficients and applies DiGA to attention and MLP projection matrices, merging other parameters with Task Arithmetic.Baseline coefficients are fixed before test evaluation, with no tuning on test sets; decoding and evaluation follow benchmark protocols.Source: A EXPERIMENTAL SETUP · p. 15

A.1 LANGUAGE-ONLY EXPERTS

  • Language-only evaluation covers five Llama-3.1-8B LoRA or OFT experts and five fully fine-tuned Llama-3.2-3B experts across specialized capabilities.The LoRA/OFT suites include five target-task benchmarks and two transfer benchmarks; the full-fine-tuning suite evaluates five capabilities using task-specific benchmarks.Source: §A.1 · p. 15

A.2 VISION-LANGUAGE EXPERTS

  • The vision-language evaluation merges three Qwen2.5-VL-7B-Instruct-based experts for spatial reasoning, OCR, and medical multimodal reasoning.Performance is reported as the macro average across five benchmarks.Source: §A.2 · p. 15

B BENCHMARKS

The benchmarks cover mathematical reasoning, instruction following, science, commonsense and social reasoning, exam generalization, safety, coding, OCR, spatial understanding, and medical multimodal reasoning.

B BENCHMARKS

  • MATH500 and GSM8K assess mathematical reasoning, with GSM8K evaluated using 8-shot chain-of-thought prompting and exact-match accuracy.MATH500 is derived from MATH and contains challenging competition-level problems; GSM8K contains grade-school word problems requiring multi-step arithmetic.Source: B BENCHMARKS · pp. 15–16
  • AGIEval tests out-of-domain generalization across exam domains in Chinese and English, while ARC-Avg averages accuracy over 12 translated ARC-Challenge variants.ARC-Challenge consists of grade-school multiple-choice science questions.Source: B BENCHMARKS · p. 16
  • IFEval measures verifiable instruction following, while ScienceQA, CommonsenseQA, and Social-IQA cover science, commonsense, and social reasoning.ScienceQA uses image-free test examples without hints, lectures, or explanations; Social-IQA asks about people’s intentions, reactions, and motivations.Source: B BENCHMARKS · p. 16
  • Safety evaluation spans harmful-response assessment, attack success, appropriate refusal of unsafe prompts, and compliance with safe requests.The supplied benchmarks include WildGuardTest, HarmBench, XSTest, and DoAnythingNow; coding uses HumanEval+ and MBPP+ for execution-based functional correctness, while OCRBench, MMSI-Bench, EmbSpatial-Bench, MMMU-medical, and PathVQA cover multimodal capabilities.Source: B BENCHMARKS · p. 16
  • Vision-language benchmarks include OCR, multi-image spatial reasoning, embodied spatial relationships, medical multimodal reasoning, and pathology-image question answering.PathVQA reports accuracy on its yes/no subset; OCRBench covers text recognition and several forms of visual question answering and information extraction.Source: B BENCHMARKS · pp. 16–17

C MERGING METHODS

The section describes five data-free merging baselines, ranging from linear task-vector addition to methods that address interference or preserve update geometry.

C MERGING METHODS

  • Task Arithmetic represents each task as the parameter difference from its base model and merges task vectors by linear addition.It serves as a simple additive baseline.Source: C MERGING METHODS · p. 17
  • TIES-Merging trims redundant parameters, elects the sign with the greatest total magnitude, and merges sign-consistent updates.The method is designed to reduce destructive interference among task-specific updates.Source: C MERGING METHODS · p. 17
  • DARE randomly drops parameters from each task vector, rescales the retained updates to preserve expected magnitude, then merges the sparse vectors.Source: C MERGING METHODS · p. 17
  • TSVM decomposes matrix-structured task updates into singular vectors and selects and combines task-specific components.Its stated aim is to reduce interference while preserving important update directions.Source: C MERGING METHODS · p. 17
  • OrthoMerge maps orthogonal transformations to the Lie algebra and merges there on the orthogonal-group Riemannian manifold.The approach is motivated by preserving the intrinsic geometric structure of pretrained weights.Source: C MERGING METHODS · p. 17

D EVALUATION

The appendix specifies evaluation protocols and the geometry, decomposition, and aggregation procedure underlying DiGA. It also addresses wide matrices and numerical handling.

D EVALUATION

  • Evaluation uses established harnesses, greedy decoding or likelihood scoring as appropriate, and benchmark-specific procedures for coding, safety, and vision-language tasks.The experiments fix expert orderings within each setting and use NVIDIA RTX A6000 GPUs with 48 GB memory per GPU.Source: D EVALUATION · p. 17

E.1 WITHIN-FRAME REORIENTATION AND THE ORIENTATION COMPONENT

  • First-order within-frame reorientation directions have the form QA with skew-symmetric coefficients, and the orientation component projects an update’s in-frame coordinates onto that skew-symmetric part.For coordinates G_i = Q^TΔ_i, the closest skew-symmetric matrix gives the projection onto the orientation subspace.Source: §E.1 · p. 18

E.2 CHARACTERIZATION OF THE STRUCTURAL SUBSPACE

  • The structural subspace is the orthogonal complement of the orientation subspace, characterized by symmetric in-frame coordinates and changes outside the reference frame’s column space.Source: §E.2 · pp. 18–19

E.3 EXISTENCE, UNIQUENESS, AND CROSS-COMPONENT ORTHOGONALITY

  • Every task update has a unique decomposition into orientation and structural components, and the component families are orthogonal across experts sharing the reference frame.The construction splits G_i into skew-symmetric and symmetric parts, assigning the out-of-frame remainder to the structural component.Source: §E.3 · p. 19

E.4 WIDE-MATRIX CASE

  • For full-row-rank wide weights, DiGA builds the shared frame from the row space and transposes the component formulas back to the original update shape.The projection, uniqueness, and cross-component orthogonality results carry over by preservation of the Frobenius inner product under transposition.Source: §E.4 · p. 20

F.1 MERGING PROCEDURE

  • Algorithm 1 forms each expert delta as Δi = Wi − W0, constructs Ti using a QR-based projection and skew component, and defines the remainder Ri = Δi − Ti.The Gram–Schmidt tolerance is 10^-8; all measured residual norms exceed 10^-1, and no direction is removed by the tolerance.Source: §F.1 · p. 20

F.2 SENSITIVITY TO EXPERT ORDERING

DiGA’s aggregate target-task results remain relatively stable across expert orderings, with most permutations outperforming OrthoMerge. An order-independent variant also exceeds OrthoMerge on Task Avg.

F.2 SENSITIVITY TO EXPERT ORDERING

  • 118 of 120 orderings exceed OrthoMerge on Task Avg.; across orderings, Task Avg. is 58.03±0.31 (range 57.04–58.67) and Transfer Avg. is 40.64±0.23 (range 40.27–41.11).Ordering sensitivity is more visible on MATH500 and ScienceQA, but diminishes when averaging across the five target tasks.Source: §F.2 · p. 21
  • Symmetric orthogonalization yields Task Avg. 58.23 and Transfer Avg. 40.71, compared with OrthoMerge’s 57.32 and 40.76.This order-independent reference retains the same component-wise aggregation procedure.Source: §F.2 · p. 21
  • The appendix also presents layer-wise OFT component similarities and LoRA geometric effects, alongside performance comparisons across listed benchmarks.Figure 4 compares orientation and structural component cosine similarities; Figure 5 reports LoRA MHE departure and expert-relative angular distortion.Source: Table 8 · p. 21, Figure 4 · p. 22, Figure 5 · p. 22

Related papers

Loading 2609.37564v1…