PaperlayerView exact revision on arXiv

Source-linked AI summary

CORAM: Coherent Orthogonal Rotation for Model Merging

Xinyi Sui, Ziran Liu, Nam Ling, Wei Wang, Wei Jiang

arXiv:2608.17366v1cs.LG

TL;DR

Model merging lacks a geometry-aware way to combine finetuning updates, while matrix-wide transforms cannot capture heterogeneous changes. CORAM merges sliced SVD factors on their natural manifolds and improves over OrthoMerge by 0.25–1.35 score points across four heterogeneous suites.

  • Problem

    Existing Euclidean task-vector arithmetic ignores update geometry, while matrix-wide transformations can mix strongly updated and unchanged directions.

  • Method

    CORAM partitions target matrices into row slices, represents expert slices in base-model SVD frames, and averages task-specific factors on corresponding manifolds.

  • Results

    0.25–1.35 score points: CORAM improves over OrthoMerge across four suites spanning three model families, 3B–9B scales, and language and vision-language tasks.

  • Takeaways & Limitations

    CORAM matches or exceeds the strongest weight-space baselines without requiring a per-benchmark sweep over λ.

  • Takeaways & Limitations

    κ values are empirical, with κ ≈0.5 calibrated on a single high-D base, and a full contraction analysis remains open.

Abstract

from arXiv · show

Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update. Orthogonal Model Merging (OrthoMerge) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base-model SVD frame, and merges the task-specific factors on their corresponding manifolds. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficient $λ=κ\hat{c}$. The scale c_hat is estimated from the expert and merged update norms and is approximately $\sqrt{N}$ for $N$ experts with comparable update magnitudes. The restoration strength kappa is selected from the dispersion of expert updates without evaluating candidate merged models. This rule remains within 0.72 points of the best swept value on all evaluated suites. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non-target layers. Across four suites covering three model families, 3B to 9B scales, and language and vision-language experts, CORAM improves over OrthoMerge by 0.25 to 1.35 points and matches or exceeds the strongest weight-space baselines.

Introduction

CORAM addresses the limitations of matrix-level Euclidean model merging with slice-level manifold operations that preserve per-slice geometry. It also introduces search-free amplification and refinements for distributing concentrated updates and restoring residual layers.

  • Slice-level manifold merging: CORAM partitions target weight matrices into row slices, represents expert slices through base-relative SVDs, and merges rotations, spectral shifts, and right factors on their natural spaces.Rotations use a log-Euclidean mean, spectral shifts use linear averaging, and right factors are combined on the Stiefel manifold.
  • Search-free amplification: Manifold averaging contracts merged updates toward the base model, so CORAM uses λ = κĉ, estimating contraction from update norms and selecting restoration strength via update dispersion.Neither coefficient requires evaluating candidate merged models; ĉ is approximately √N for N comparable-magnitude experts.
  • Geometry-motivated refinement: Spread slicing distributes highly updated rows across slices, while a residual pathway restores fine-tuning updates from layers outside the sliced merging targets.These refinements address concentrated updates and adapt residual decoupling to layer-level residuals.
  • Evaluation: CORAM is evaluated across four suites, three model families, 3B–9B model sizes, and both language and vision-language experts.The evaluation covers heterogeneous model-merging settings.
  • Search-free amplification: 0.72 score points: the predicted amplification coefficient remains within the swept optimum on every evaluated suite.The rule combines contraction-scale estimation from expert and merged update norms with zero-evaluation selection of κ.

Related Work

Prior model-merging methods combine task vectors in Euclidean weight space, while newer approaches align units, use tangent spaces, or construct SVD-based subspaces. CORAM is closest to OrthoMerge and relates its merge coefficient to geometric contraction, using update norms and expert dispersion instead of benchmark-specific sweeps.

  • Euclidean weight-space merging: Euclidean merging combines finetuned experts as task vectors, with methods differing mainly in update weighting or interference reduction.Uniform averaging, Fisher- or regression-weighted merging, task arithmetic, TIES, DARE, and Localize-and-Stitch exemplify these approaches.
  • Geometric and subspace merging: Geometric methods align model units, merge in tangent spaces, or construct SVD-based subspaces from task updates.OrthoMerge averages one orthogonal transform per weight matrix and merges the remaining residual conventionally.
  • Coefficient selection in model merging: Coefficient-selection methods use fixed constants, benchmark-selected values, test-time per-layer optimization, or geometric rules for same-task finetuning.Task arithmetic, benchmark protocols, AdaMerging, and Model Stock represent these alternatives.
  • Coefficient selection in model merging: CORAM relates the coefficient to geometric-merge contraction and selects restoration strength from expert-update dispersion without a per-benchmark sweep over λ.Contraction is measured from expert and merged update norms, motivated partly by base-model-dependent merging behavior.

Method

CORAM partitions target matrices into row slices, represents expert slices in the base model’s SVD frame, and merges their structured factors on native manifolds. It then handles conflicting columns and restores contraction-induced magnitude loss using evaluation-free amplification and dispersion-based restoration strength.

  • Slice-wise representation: CORAM partitions each target matrix into row slices and represents each expert slice by a rotation, relative spectral shifts, and a right factor in the base SVD frame.The factors lie on SO(h), structured spectral-shift space, and the Stiefel manifold St(din, h).
  • Manifold merging: CORAM averages rotations with the log-Euclidean mean, spectral shifts linearly, and right factors with a polar mean on the Stiefel manifold before reconstructing and stacking slices.The reconstructed slices produce Wmerge for each target matrix, yielding the geometric model merge.
  • Conflict handling: Conflict-aware variants detect per-slice, per-column updates opposing consensus, then discard flagged components or return them through a residual pathway.Three variants combine manifold merging and residual return differently; the unused fourth would discard the signal on which experts agree.
  • Contraction restoration: Manifold averaging preserves the merged direction but contracts its magnitude, so CORAM applies a global amplification coefficient λ = κ c_hat determined without evaluating candidate merged models.The contraction scale is measured from typical expert and merged update magnitudes, while κ controls restoration strength.
  • Evaluation-free coefficient selection: Across four low-dispersion settings, cRMS = 2.09/2.03/1.72/2.09, compared with N = 2.24/2.24/1.73/2.24, with a difference below 10% in each case.CORAM therefore uses λ = κ N by default, while selecting κ ≈1.15 for low dispersion and κ ≈0.5 for high dispersion from cached update statistics.

Geometry-Derived Refinements

CORAM adds optional spread slicing and a residual pathway to address uneven update distributions and non-target layers. Spread slicing redistributes important rows before geometric merging, while residual updates are merged conventionally and added as a separate patch.

  • Refinement combinations: The refinements are optional: spread slicing works alone when update magnitudes are comparable, but pairs better with the residual pathway when magnitudes differ substantially.Their effects are evaluated separately and together and depend on the distribution of expert-update magnitudes.
  • Spread slicing: Spread slicing distributes heavily updated rows more evenly across slices, avoiding poorly conditioned SVDs and slices with little update signal.Rows are scored by relative update magnitude across experts before assignment, then the permutation is inverted after merging.
  • Spread slicing: Four row-priority rules are provided: mean, energy, variance, and owner balance.Owner balance favors rows dominated by one expert that also have large updates, assigns them to the least-loaded slice, and breaks concentration ties.
  • Residual pathway: The residual pathway merges embedding and normalization updates omitted by slice geometry using task arithmetic, TIES, or Task Singular Vectors, then adds the result as a separate patch.This configuration is denoted CORAM+RP.
  • Residual pathway: Only the geometric merge is amplified, because manifold averaging contracts it while the Euclidean residual merger needs no correction.With both refinements enabled, CORAM+SS+RP applies spread slicing before geometric merging and adds the unscaled residual patch afterward.

Experiments

Experiments across four suites show that CORAM consistently improves over OrthoMerge, while the search-free κ-rule stays close to the best swept λ without per-suite evaluation sweeps. Ablations further identify when conflict awareness, spread slicing, and the residual pathway help or hurt.

  • Cost: Slice SVDs are computed once and cached, while each geometric merge costs 0.5 to 0.8 GPU-hours per configuration.Sweeping λ costs one to two orders of magnitude more than the merge itself.
  • Main comparison: CORAM beats every OrthoMerge variant by +1.08 on T1, +1.35 on T2, +0.82 on T3, and +0.25 on T4.The comparisons use a single evaluation harness and a reproduction of OrthoMerge.
  • κ-rule: The search-free κ-rule comes within 0.72 points of the swept-λ optimum across all suites and configurations.It avoids per-suite sweeps of ten or more λ grid points, each requiring full-suite evaluation.
  • κ-rule: κ separates low- and high-dispersion cases: D ∈[1.3, 3.5] uses κ approximately 0.9–1.2, whereas Gemma-2-9B has D ≈16 and selects κ≈0.5.The dispersion value is computed before task evaluation, but the experiments do not establish κ behavior for intermediate D.
  • Ablations: CORAM-C improves over plain CORAM by +0.21 on T2 and +0.07 on T3 under the κ-rule.Combining spread slicing and the residual pathway gives the best selected-κ values on T2 and T4: 43.48 and 54.71.
  • Ablations: On T4, spread slicing alone is 0.58 to 0.86 points below its contiguous counterpart when update magnitudes differ substantially.The reduction is concentrated in safety, which decreases by 4.7 points; the residual pathway restores omitted embedding and normalization updates.

Limitations

CORAM’s limitations include untested intermediate D values and restricted evaluation scope. Its scale and restoration rules rely partly on empirical calibration, while theoretical explanations for contraction and spread-slicing failures remain incomplete.

  • Evaluation coverage: Intermediate D values are not extrapolated because current public expert suites contain only four low-D bases and one high-D base.Intermediate values are not represented in the evaluated suites.
  • Scale and strength: The ideal scale is theoretically motivated, but cRMS extends it to correlated or uneven updates and remains within 10% of N when update magnitudes are comparable.The passage attributes the ideal scale to Proposition 1 and the extension to cRMS.
  • Scale and strength: The restoration values κ ≈1.15 and κ ≈0.5 are empirical D-selected values, with the latter calibrated on a single high-D base.A full contraction analysis of the composed slice map remains open.
  • Behavioral evidence: Spread slicing can fail under uneven updates, although the residual pathway corrects the observed domain- and generation-level failures; no theoretical explanation is provided.The limitation is behavioral rather than theoretically explained.
  • Scope: The evaluation follows MergeBench and Yang, Shi, and Liu (2026) protocols and excludes larger expert sets, mixture-of-experts architectures, and experts from different base checkpoints.These settings are outside the reported evaluation scope.

Conclusion

CORAM merges finetuned experts at the slice level by representing each slice in the base model’s SVD frame and averaging task-specific factors on corresponding manifolds. It offsets manifold-induced contraction with norm-based amplification and selects restoration strength from update dispersion without evaluating candidate merged models.

  • CORAM represents each expert slice with its SVD in the base-model frame and averages task-specific factors on corresponding manifolds.
  • CORAM compensates for contraction from manifold averaging using an amplification scale estimated from expert and merged update norms.
  • CORAM selects restoration strength from update dispersion without evaluating candidate merged models.

Supplementary Material for CORAM: Coherent Orthogonal Rotation for Model Merging

The supplement expands CORAM with complete method details, experimental results, and analyses omitted from the main paper. It covers the pipeline, amplification diagnosis, and evaluation protocol.

  • The supplement provides method details, complete experimental results, and additional analyses omitted from the main paper.
  • Section A: Section A covers the CORAM pipeline, sign alignment, geometric operators, conflict-aware variants, spread slicing, and the residual pathway.
  • Section B: Section B presents the branch diagnosis motivating amplification, Proposition 1, and measurements of contraction scale and restoration strength.
  • Section C: Section C describes the evaluation protocol, computing infrastructure, baselines, configuration naming, and mapping to the main tables.

A. Additional Method Details

CORAM’s implementation decomposes sliced expert matrices in the base-model SVD frame, merges rotation, spectral-shift, and right-factor components geometrically, and amplifies the reconstructed update. Additional mechanisms provide spread slicing, conflict-aware residual alternatives, and separate residual merging for non-target parameters.

  • Complete pipeline: Each target matrix is row-sliced, factorized in the corresponding base-model SVD frame, merged across three factor spaces, reconstructed, and then amplified.The merged factors are the rotation, relative spectral shift, and right factor.
  • Numerical audit: Maximum condition numbers fall from 706 to 38, 822 to 28, 635 to 282, and 43 to 18 with variance-spread slicing.The audit reports unchanged median condition numbers and tail collapse, with no slice having σmin < 10^-8.
  • Spread slicing: Spread slicing ranks rows by relative update magnitude, distributes highly updated rows across slices, and restores the original order after merging.The same permutation is applied to the base and all experts, followed by its inverse after merging.
  • Conflict-aware variants: Three conflict-aware variants assign flagged and non-flagged columns to geometric merging, discarding, or Euclidean residual averaging; plain CORAM uses no mask.The fourth combination is excluded because it removes the signal on which experts agree.
  • Residual pathway: For non-target parameters, CORAM separately merges updates with task arithmetic, TIES, or Task Singular Vectors and adds the result as an unscaled residual patch.The geometric merge targets attention and MLP linear layers, while amplification applies only to the geometric update.

B. Amplitude Restoration … Mapping to the Main Tables

CORAM’s amplitude restoration is motivated by directional cancellation and contraction in manifold-merged updates, while its evaluation uses deterministic, suite-specific protocols and explicitly mapped configurations. The supplementary sections define the restoration rule, evaluation settings, baseline handling, and configuration conventions behind the main tables.

  • B. Amplitude Restoration: Only branches containing both rotations recover the task direction, while the merged update contracts to 0.21 ∥∆TA∥ at λ=1.Spectral shift alone is inert, and left rotation alone is task-orthogonal; the right factor alone recovers only part of the signal.
  • B. Amplitude Restoration: The cancellation analysis motivates √N amplification for incoherent expert updates, with cRMS measuring contraction of the complete reconstructed update.The equal-norm result extends to unequal norms, so norm imbalance alone does not change the cancellation scale.
  • B. Amplitude Restoration: For low-dispersion bases, useful κ values are approximately 1.0–1.2, whereas Gemma-2-9B uses near 0.5 at D ≈16.The smaller high-dispersion restoration strength is attributed to correlated updates and nonlinear slice-map effects, with D serving as an empirical proxy.
  • B. Amplitude Restoration: The evaluated bases form two separated dispersion groups, and no interpolation is defined for intermediate D values.The four low-dispersion settings have D ∈[1.3, 3.5], while the high-dispersion Gemma-2-9B setting has D ≈16.
  • Evaluation Protocol: T1 merges five Llama-3.1-8B experts across five in-domain tasks and evaluates M-ARC and AGIEval out of domain under a shared zero-shot setting.The in-domain tasks are MATH500, HumanEval+, ScienceQA, CommonsenseQA, and Social-IQA.
  • Evaluation Protocol: T2 and T4 merge five Llama-3.2-3B or Gemma-2-9B experts across instruction following, mathematics, coding, multilingual understanding, and safety.T3 instead merges three Qwen2.5-VL-7B-Instruct vision-language experts and evaluates MMSI-Bench, EmbSpatial, MMMU-Med, PathVQA, OCRBench, and CharXiv.
  • Evaluation Protocol: Evaluations use fixed task-specific harnesses, and merging is deterministic; reported scores are single runs unless the multi-seed study applies.Language uses lm-evaluation-harness, code uses the bigcode evaluation harness, vision-language uses lmms-eval, and safety follows the MergeBench safety stack.
  • Baselines and Mapping to the Main Tables: Linear averaging, task arithmetic, TIES, and DARE-TIES follow their original per-tensor definitions, while OrthoMerge reports its strongest released variant under the same harness.The residual pathway is not applicable to T1 because its released OFT experts modify only target linear layers; Table 5 records supplementary configurations for main-table CORAM rows.

D. Complete Configuration Results and Coefficient Sweeps … Alternative Coefficient Choices

The supplementary results document per-suite coefficient sweeps, broad configuration-grid comparisons, and validation of the κ-rule against swept optima. They also establish stable dispersion-based κ choices and describe alternative coefficient evaluations, with T3 using five-task averages that exclude CharXiv.

  • D. Complete Configuration Results and Coefficient Sweeps: The retained configurations report swept-optimal coefficients and corresponding in-domain averages, while T3 sweeps use five deterministic non-judged tasks excluding CharXiv.Main-table summaries and final checkpoint comparisons instead include CharXiv and use six-task averages unless stated otherwise.
  • Sweep Curves: The κ-rule coefficient loses at most a fraction of a point because in-domain curves are flat around their optima.Figure 3 compares representative and strongest-branch lines across the four suites against κ-rule and swept coefficients.
  • Complete Rule-versus-Optimum Comparison: The rule-versus-optimum comparison covers every retained line, with 50 of 52 rule-evaluated lines satisfying the 0.72-point bound.The two exceptions are exploratory combinations not selected for main-table rows: lr.ties on T3 with a 0.89 gap and nca+ave on T4 with a 0.73 gap.
  • The Complete Configuration Grid: The development grid evaluated 238 configuration lines and 1,645 (line, λ) points across conflict, spread, residual, and combined spread-plus-residual variants.Table 20 separates κ-rule-selected results from best values found by sweeping λ.
  • Distribution of the Swept Optima: On T4, swept values span approximately κ ∈[0.45, 0.58], motivating the rounded shared high-dispersion constant κ = 0.5.For the low-dispersion case, the single swept Gemma-2-2B configuration has κopt ≈ 1.07; κ = 1.15 maps to λ = 2.60 for N = 5 and λ = 2.00 for N = 3.
  • Threshold Sensitivity: Every threshold D0 in the open interval (3.5, 16) produces the same κ assignment, selected coefficients, and rule-to-optimum gaps.Low-dispersion bases have D ∈[1.3, 3.5], while the single high-dispersion base has D ≈16.
  • Leave-One-Base-Out Validation: Leave-one-base-out validation re-estimates κ from the other low-dispersion bases and tests the resulting predicted coefficient on each held-out base.The held-out estimate ˆκ is the mean over the other three bases, including Gemma-2-2B, with N rounded to the nearest evaluated grid point.
  • Alternative Coefficient Choices: The paper also tests several closed-form alternatives over λ ∈[1, 4] and summarizes their predictions and observed limitations.The supplementary material states that complete machine-readable sweep logs will accompany the released artifacts.

E. Component Ablations … OrthoMerge Reproduction

The ablations isolate conflict handling, spread slicing, and residual restoration, while additional analyses examine T4 safety degradation, cross-suite transfer, update retention, and slice height. OOD comparisons and OrthoMerge re-evaluation contextualize CORAM’s suite-level performance and protocol differences.

  • E. Component Ablations; Conflict-Aware Merging; Spread Slicing; Residual Pathway; Interaction between Spread Slicing and the Residual Pathway: Matched ablations separately evaluate conflict handling, spread slicing, the residual pathway, and their interaction while fixing other configuration choices.The component study reports branch-level κ-rule results and matched comparisons for each mechanism.
  • Additional Analysis of Spread Slicing on T4: −6.29 on Safety versus +0.04 on Multilingual in one matched spread comparison; across 49 pairs, safety decreases range from 3.2 to 7.3 points, with median 4.9.The difference is concentrated in safety, especially over-refusal accuracy on safe prompts, while adversarial performance remains unchanged.
  • Additional Analysis of Spread Slicing on T4; Layer-Wise Update Retention: 137 spread-only responses versus 24 contiguous responses were classified as refusals on 250 safe prompts, yielding safe-side accuracy 0.452 versus 0.904.Generation inspection attributes the degradation to unstable open-ended output and mixed-script drift rather than truncation or silence.
  • Additional Analysis of Spread Slicing on T4; Interaction between Spread Slicing and the Residual Pathway: Adding the residual pathway raises the T4 safety score from 51.31 for spread-only merging to 56.63, while the combined in-domain average is 54.71 at the κ-rule point.The residual pathway partially repairs spread-induced instability; the residual-only in-domain average is 54.56.
  • A Single Configuration Across All Suites: A frozen plain CORAM configuration trails per-suite best results by 0.22–0.73 points, exceeds OrthoMerge on T1–T3, and remains within 0.25 on T4.The configuration uses contiguous slicing, h = 8, all three factors, no conflict masking, no spread slicing, no residual pathway, and κ-rule λ.
  • F. Slice-Height Study; Complete Slice-Height Comparison: No h = 16 point matches its suite’s h = 8 optimum; the largest reported advantage for h = 16 is 0.11, while T3 h = 64 collapses to 0.0 accuracy.The κ-rule coefficient is reported for h ∈{8, 16, 32, 64}, with other settings fixed within each comparison.
  • G. OOD Evaluation and OrthoMerge Reproduction; In-Domain versus Out-of-Domain Trade-off; OOD Evaluation Differences; OrthoMerge Reproduction: CORAM attains the highest in-domain average on T1–T3, trails OrthoMerge by 0.12 on T4 at the rule point, and exceeds it when swept.Its OOD averages remain within the band of the mildest mergers on every suite; reproduced OrthoMerge scores are 42.13 versus 42.07 on T2 and 63.56 versus 64.04 on T3.

H. Multi-Seed Results · I. Computational Cost

Multi-seed re-evaluation shows that CORAM’s deterministic merges produce stable scores except where coding samples and the CharXiv judge introduce variation. Computationally, cache reuse and the κ-rule reduce repeated evaluation costs, although CORAM construction is costlier than per-tensor baselines.

  • H. Multi-Seed Results: Multi-seed comparisons repeat one CORAM checkpoint and its closest baseline across the original run and four additional decoding seeds.The additional seeds are 18, 38, 68, and 98.
  • H. Multi-Seed Results: Merging is bitwise reproducible, while seed variation comes only from coding samples and the CharXiv external judge.All remaining tasks produce bit-identical scores across the four re-evaluation seeds.
  • H. Multi-Seed Results: Safety confidence intervals are estimated for the two closest comparisons using paired bootstrap, conservative binomial errors, and delta-method combination.HarmBench and WildGuard use 2,000 paired stratified bootstrap resamples, while DAN and XSTest use unpaired binomial errors.
  • I. Computational Cost: Slice-SVD caches are constructed once and reused across configurations and coefficient values.The cost accounting covers cache construction, geometric merging, coefficient assembly, and evaluation.
  • I. Computational Cost: 4,000 GPT-judge requests make each CharXiv evaluation cost approximately $10 per pass, while a T1 full merge takes 12 minutes on a B300-class GPU.Each CharXiv pass covers 1,000 figures and four descriptive questions per figure.
  • I. Computational Cost: CORAM costs more to construct than per-tensor baselines, but merging remains small relative to full evaluation and the κ-rule removes repeated λ-sweep evaluations.The κ-rule’s main saving is avoiding candidate-model evaluations during coefficient selection.
  • I. Computational Cost: Evaluation-dominated sweeps total 17–20 GPU hours on T2 and 40–48 GPU hours on T4.The T3 evaluation includes five VL tasks at 0.8 GPU hours and one CharXiv pass at about 0.6 hours.
Loading 2608.17366v1…