GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment

• Prior merging methods often compress all tasks into one shared space, causing disproportionate degradation for incompatible tasks, and some require private calibration data or activations [Ilharco et al., 2022, Yadav et al., 2023].Source: §1 · pp. 1–2
• Higher intra-coalition subspace similarity predicts greater merged-model accuracy and lower projection error, while geometrically isolated outlier tasks benefit from group-wise isolation.Figure 1 illustrates the accuracy–projection-error trade-off across partitions; the introduction identifies cluster-outlier structures in GLUE and block-diagonal affinities in vision.Source: §1 · p. 2, Figure 1 · p. 2
• For task-aware deployment, GRAM trades a marginal increase in storage for compressing T models into K ≪ T models and requires only fine-tuned weights, not source data, activations, or calibration sets.This addresses privacy-constrained settings such as healthcare, where institutions cannot share local clinical data.Source: §1 · pp. 1–2
• GRAM partitions tasks with Grassmannian k-means++ on chordal distances, then uses a greedy adaptive rank allocator to pool singular values within each coalition under a global rank budget.The rank allocator is claimed to achieve provable optimality under that budget.Source: §1 · p. 2
• The clustering objective yields merge error within O(log K) of the optimal partition, with expected error strictly below that of a standard global merge.Source: §1 · p. 2
• GRAM consistently outperforms existing data-free baselines across language, vision, and medical instruction-tuning, and surpasses calibration-data-dependent methods on the medical benchmark.The cited medical methods are RegMean and Fisher-Merge.Source: §1 · p. 3

Model Merging

• Parameter-level, coefficient-learning, and structured-update methods combine fine-tuned weights or task vectors, learn mixing weights, or exploit update structure [Wortsman et al., 2022, Ilharco et al., 2022, Yadav et al., 2023, Yu et al., 2024, Yang et al., 2023, Tang et al., 2023, Gargiulo et al., 2025, Stoica et al., 2024, Marczak et al., 2025, Chen et al., 2025b, Wei et al., 2025, Wang et al., 2024b, Sun et al., 2025, Fu et al., 2025].Multi-model approaches retain some per-task structure rather than collapsing to one shared model [Huang et al., 2024].Source: §2 · p. 3
• Single-model merging methods share a global merge that assumes sufficient subspace alignment, while this work measures task subspace geometry and merges compatible groups.The assumption can fail when tasks span diverse domains.Source: §2 · p. 3

Task Grouping in Multi-Task Learning

• Prior task-grouping methods use transfer relationships, partitions, task representations, architectures, or affinity signals, but require training-time data and gradients; this criterion uses only task-update left singular subspaces.Examples include TAG [Fifty et al., 2021], MTG-Net, Grad-TAG [Li et al., 2024], and Wang et al. [2024a]; dynamic loss weighting [Liu et al., 2022] and client cooperation [Yang et al., 2026] address competition or interference without explicit task partitioning.Source: §2 · p. 3

Task vectors

• Task vectors are fine-tuned-minus-pretrained weight updates; the matrix formulation covers merged linear layers, with non-matrix vision parameters averaged separately.For LoRA, task vectors inherit a rank-r factorization. The task-vector view follows prior work [Ilharco et al., 2022, Yadav et al., 2023, Yu et al., 2024].Source: §3 · p. 3

Merging objective

• The objective is to assign T rank-r task vectors to K<T groups and minimize total reconstruction error under a rank-r budget for each merged update.Each task is served by the model formed from the pretrained backbone plus its assigned group’s merged task vector.Source: §3 · p. 4

Subspace interference

• A task’s projection error depends on its output singular vectors and singular values, not its input singular vectors; without singular-value weighting, the error is squared chordal distance between subspaces.The principal-angle relation is a standard result [Golub and Van Loan, 1996].Source: §3 · p. 4
• When task and shared subspaces align, projection error vanishes; misalignment discards important task directions, motivating grouping by subspace compatibility before merging.Figure 2 depicts SVD extraction, chordal-distance clustering into K coalitions, and thin-SVD construction of a rank-r update within each coalition.Source: §3 · p. 4, Figure 2 · p. 5

4 Method: Group-wise Rank-Aware Model Merging

• GRAM groups task vectors into K coalitions using pairwise chordal subspace geometry, then performs a low-rank merge within each group to limit interference.The pipeline uses an SVD-based merge within each coalition.Source: §4 · p. 5

4.1 Chordal Subspace Distance

• Chordal distance directly governs the projection error incurred when tasks share a basis, making it a natural grouping metric rather than flattened-coordinate distance.Source: §4.1 · p. 5, §4 · p. 5

4.2 Grassmannian k-Means++ Clustering

• Layer-averaged chordal distances feed Grassmannian k-means++ to form K groups; K is user-set, and K=2 gave the best cost–accuracy trade-off across experiments.Source: §4.2 · p. 6

4.3 Per-Group Low-Rank Merge

• Within each coalition, GRAM allocates rank by selecting the largest remaining squared singular values across tasks, minimizing total reconstruction error under the global rank budget.It concatenates the selected SVD components, globally sorts them, and orthonormalizes with polar factors to reconstruct each group update; adaptive allocation addresses sharply varying intrinsic ranks across tasks and layers.Source: §4.3 · p. 6
• With optimal per-group bases, any partition into K groups has total reconstruction error no greater than a single global rank-r basis fit to all tasks.The guarantee is independent of the clustering rule; the text says gains are largest for tasks geometrically furthest from the bulk.Source: §4.3 · p. 6

5.2 Language Understanding

• 83.8% average on Flan-T5-Base (+2.1 over TSV-Merge’s 81.7%) and 88.7% on Flan-T5-Large (+0.9 over TSV-Merge’s 87.8%) with K=2.MNLI and STS-B recover most relative to TSV-Merge at both scales; on QQP, GRAM reaches 84.0 on Base.Source: §5.2 · p. 7

5.3 Vision

• 88.7% B/32 and 94.0% L/14 average accuracy: GRAM surpasses Iso-CTS (85.8% / 93.9%) and TSV-Merge (85.0% / 92.8%).The +2.9 B/32 gap to the next-best baseline accompanies strongest results on GTSRB (94.6 / 98.3) and EuroSAT (99.1 / 99.3), tasks distant from the visual-recognition cluster.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 Medical Instruction Tuning

• +75.54% Avg Impr. and 2.40 Avg Rank: GRAM leads all merging methods, ahead of data-using RegMean (+71.48%, 3.40).Its gains concentrate on open-ended generation metrics; it trails the leader by under one point on the two MCQA accuracy columns. Its summary-metric edge also exceeds the Individual oracle (+69.62%, 4.55).Source: §5.4 · pp. 8–9, Table 3 · p. 8

5.5 Ablation Studies

• Removing PA-Sim, Coalition, or Adapt-Rank lowers performance; the largest drop is PA-Sim on GLUE (−3.0) and Coalition on ViT-B/32 (−3.4).Coalition ablation collapses GRAM to K=1; adaptive rank allocation matters more for heterogeneous LoRA spectra in PEFT than for full-fine-tuned vision deltas.Source: §5.5 · p. 9, Table 4 · p. 9
• At K=2, GRAM gains +5.6 Base and +2.4 Large points over Weight Averaging, capturing 87% and 73% of the respective gaps to the no-merging oracle.Raising K from 2 to 7 adds at most +0.1 Base and +0.6 Large points; Base average dips at K=4, so most gains are available with one extra rank-16 adapter.Source: §5.5 · p. 9
• GRAM’s partition is stable across 10 initialization seeds at K=2 and K=3 (10/10), and agglomerative average linkage reproduces it at K ∈ {2, 3, 5, 6, 7}.At K=2, degree-corrected modularity instead chooses a 4/4 split that scores 0.9 lower (82.9); GRAM uses the stronger 6/2 split.Source: §5.5 · pp. 9–10, Table 5 · p. 9
• A task mapper routes 100.0% of Flan-T5-Base / GLUE validation examples correctly at K=2, making task-agnostic GRAM equal oracle GRAM (83.8 on every task).Routing remains ≥99.98% for every K; grouping takes about one second for 8 tasks and finishes within a few minutes for 1024 tasks, with time and memory growing roughly linearly.Source: §5.5 · p. 10, Figure 3 · p. 10

6 Conclusion

• With K=2 (one extra deployed model), GRAM matches strongest GLUE baselines (83.8%/88.7% average), leads 8-task vision (88.7%/94.0%), and attains best medical Avg Rank without calibration data.The reported models are Flan-T5-Base/Large for GLUE, CLIP ViT-B/32 and ViT-L/14 for vision, and Qwen2-7B-Instruct with four LoRA experts for the medical benchmark.Source: §6 · p. 10
• GRAM groups tasks by chordal subspace distance and merges each group with SVD; gains concentrate on tasks most dissimilar from the majority, as predicted by the alignment penalty.The conclusion frames this grouping as avoiding global-merge projection loss on misaligned subspaces.Source: §6 · p. 10
• Reported scope is limited by single-seed results, one model family per modality, a public-benchmark medical case study, and theory that motivates rather than guarantees GRAM’s exact merge.Small Flan-T5-Large margins warrant caution; the theory analyzes an optimal basis and Grassmannian k-means objective rather than the exact column-stacked merge with adaptive allocation.Source: §6 · p. 10
• Future work includes scaling the task-agnostic router beyond small task collections and evaluating medical merging in a real multi-site federated setup.The cited work notes that group routing approaches full task identification for small collections [Ge et al., 2023].Source: §6 · p. 10

A Broader Impact

• GRAM combines independently fine-tuned task vectors into a deployable model without retraining or pooling raw training data, with geometry-aware grouping enabling scale beyond a handful of tasks.The paper identifies this property as carrying concrete societal value in healthcare and decentralized model merging.Source: A Broader Impact · p. 14
• In privacy-constrained healthcare, institutions can merge geometrically compatible task vectors while keeping patient data on-premises, pooling expertise across specialties without forcing dissimilar tasks into one averaged model.The passage also identifies legal, financial, and on-device personalization as privacy-sensitive settings where the same property applies.Source: A Broader Impact · p. 14
• For decentralized model development, GRAM uses a small number K of geometry-respecting merged checkpoints as a compression knob, lowering deployment costs without centralizing data or using per-adapter routing.The passage contrasts this coalition structure with naive averaging, which destroys task-specific structure, and per-adapter routing, which has linear deployment cost.Source: A Broader Impact · p. 14
• GRAM’s geometric grouping does not prevent biased or harmful behavior or value misalignment, and redistributing adapters can release unintended capabilities, so coalitions require audits and standard licensing and provenance practices.The passage recommends fairness and safety scrutiny and names adapter cards, licensing terms, and watermarking as applicable practices.Source: A Broader Impact · p. 14

Geometric setup

• For any orthonormal shared basis, task projection error equals the squared chordal distance between its learned subspace and the shared subspace.The chordal distance is expressed as a sum of squared sines of principal angles.Source: B Deferred Theoretical Analysis · p. 14

Additional theoretical results

• The deferred results also extend the analysis to GRAM’s column-stacked basis, link the partition-level error chain to a k-means++ approximation guarantee, and include computational-cost and Rademacher-complexity results.The proofs are collected in section C.Source: B Deferred Theoretical Analysis · p. 14

Theorem B.1

• Grouping tasks and fitting an optimal rank-r basis per group yields no greater total projection error than one global optimum, with strict improvement if a group’s basis improves on the global optimum for that group.The argument is a direct optimality comparison and does not depend on the clustering rule.Source: B Deferred Theoretical Analysis · pp. 14–15

Theorem B.2

• Theorem B.2 gives a direct projection-error bound for a GRAM per-group basis when task rank allocations sum to at most r and the basis contains the concatenated per-task top-k_i singular vectors.The bound covers every task in the coalition, including the stated k_i = 0 case.Source: B Deferred Theoretical Analysis · p. 15

B.1 Grassmannian k-Means++ Error Bound

• The deterministic latent-group model assumes separated prototype subspaces and task subspaces within squared chordal distance ϵ of their group prototype.It has G ≥ 2 groups, m ≥ 2 tasks per group, and equal task weights Σ_i = σI_r.Source: §B.1 · p. 15
• With equal task weights, the optimal per-group basis is the chordal Fréchet mean, and the partition error equals σ² times the Grassmannian k-means cost.The stated bounds concern this optimal-basis partition objective, not guarantees on GRAM’s actual column-stacked basis.Source: §B.1 · p. 15
• In the small-ϵ regime, k-means++ grouping is no worse in expectation than a random equipartition, while any equipartition is deterministically no worse than a single-block merge.These guarantees compare the partition objective, not GRAM’s actual column-stacked basis.Source: §B.1 · pp. 15–16

B.2 Computational Complexity

• GRAM’s dominant cost is O(TL(d_out + d_in)r²) for LoRA deltas or O(TLd_outd_inr) for full-rank deltas, both cheaper than one fine-tuning epoch.Inference costs O(dr) per layer, with no extra per-query FLOPs; storage is reduced by a factor of T/K versus storing T task models.Source: §B.2 · p. 16

B.3 Generalization via Rademacher Complexity

• For i.i.d. samples with ∥x_i∥₂ ≤ R, the Rademacher bound for the linear-update class is βR/√n.This is the bound stated for the class in Theorem B.6.Source: §B.3 · p. 16
• The per-group merged class has capacity no larger than the global class.Source: §B.3 · p. 16

C.1 Proof of Theorem B.1

• Using each group’s optimal rank-r basis yields error no greater than global merging; the inequality is strict when a group basis strictly improves on the global optimum.Otherwise, the inequality can be tight.Source: §C.1 · p. 17

Proof of Theorem B.2

• The shared projector’s column space contains each task’s truncated subspace, so projector monotonicity compares its projected energy with the task’s self-projection energy.The proof uses this comparison when applying Theorem C.1 to the task update.Source: §C.2 · pp. 17–18
• A task projected onto its own truncated left-singular subspace recovers the truncation energy, with zero energy when its retained rank k_i is zero.The proof treats k_i = 0 separately and uses orthonormality of the retained singular vectors when k_i ≥ 1.Source: §C.2 · p. 17
• For a single-task coalition, the shared and task projectors coincide, making the inequality an equality and attaining the bound.In this case, the coalition matrix is the task’s truncated left-singular basis.Source: §C.2 · p. 18
• The theorem bounds projection error onto the GRAM subspace, but not the separate in-subspace fitting error of the single merged delta against a task’s component.The passage decomposes the merged-delta error orthogonally and states that only the projection-error term is controlled.Source: §C.2 · p. 18
• The bound can be loose when other coalition tasks add directions aligned with the task’s unretained singular subspace, because the proof lower-bounds using only its smaller projector.The passage identifies this as the regime in which grouping is most useful.Source: §C.2 · p. 18

C.3 GRAM vs. global merge under uniform allocation (statement and proof)

• GRAM’s projection-error upper bound is no larger than the global-merge bound for each task under uniform allocation, with strict improvement when an added singular value is nonzero.The comparison is between upper bounds, not actual projection errors.Source: §C.3 · pp. 18–19
• The stated GRAM-versus-global guarantee concerns worst-case upper-bound gaps, not actual projection-error comparisons.The multiplicative gap between alignment-aware and random grouping is addressed by Theorem B.4, not this corollary.Source: §C.3 · p. 19

C.4 Equivalence of chordal and Euclidean distances on projectors

• A subspace’s projector provides a basis-invariant Euclidean embedding of the Grassmannian, with chordal distance equal to Euclidean distance up to a factor of 2.Source: §C.4 · p. 19

C.5 Chordal Fréchet mean and the GRAM merge basis

• The chordal Fréchet-mean subspace is a top-r invariant subspace of the sum of task projectors, unique when λ_r(M) > λ_{r+1}(M).Otherwise, any r-dimensional subspace within the top eigenspace is optimal.Source: §C.5 · p. 19
• The Fréchet-mean basis generally differs from GRAM’s column-stacked basis; the projection-error bound applies to any orthonormal basis of that column space.The latent-group analysis instead defines error using the optimal-basis projection error and invokes the Fréchet mean as the optimum, not as GRAM’s actual output.Source: §C.5 · p. 19

C.6 Proof of Theorem B.3

• For two task subspaces, the nonzero spectrum of their summed projectors is determined by principal angles, with top-r eigenvalues 1 + cos θ_j.Equality in the projection bound holds when the merge basis spans a top-r eigenspace, equivalently the top-r left singular subspace of the concatenated bases.Source: §C.6 · pp. 19–20

C.7 Proof of Theorem B.4

• E[Err(π++)] ≤ 8(ln K + 2) Err(π*) under the deterministic latent-group model.The model assumes G groups of m tasks, prototype subspaces separated by at least ϕmin, and within-group spread d2c(Ui, Qg(i)) ≤ ϵ.Source: §C.7 · p. 20
• Each mixed group contributes at least σ2[r(1 − cos ϕmin) − 4√(rϵ)] to Err(πrand).The proof uses the prototype separation and within-group spread assumptions; its bound applies when ϵ ≤ r(1 − cos ϕmin)/4.Source: §C.7 · pp. 20–21
• For global merging, the proof constructs ⌊T/2⌋ disjoint inter-group task pairs and applies a per-pair lower bound to obtain a global merge lower bound.For random equipartitions, it also analyzes the expected number of mixed groups; the supplied passages omit the resulting expression.Source: §C.7 · p. 21

C.8 Proof of Theorem B.5

• Err(πrand) ≤ Err(πglobal) holds deterministically for any partition comparison in the stated proof.The proof substitutes the global minimizing basis as a feasible, potentially suboptimal basis within each partition group.Source: §C.8 · p. 22
• When ϵ ≤ c rα2/(m ln K) for a sufficiently small absolute constant c, E[Err(π++)] ≤ E[Err(πrand)].Here α := 1 − cos ϕmin; the proof derives the comparison from Theorem B.4's bounds in the small-ϵ regime.Source: §C.8 · p. 22

C.9 Non-uniform singular values (statement and proof)

• For non-uniform singular values, Proposition C.6 gives partition-error bounds parameterized by κ := σmax/σmin.The proof relates fixed-basis error to chordal subspace error and uses feasible bases to establish the bounds; the supplied passages omit their explicit formulas.Source: §C.9 · pp. 22–23
• The k-means++ guarantee E[Φ(π++)] ≤ 8(ln K + 2)Φ(π*) is model-independent and applies to any task subspaces Ui.The proof combines the k-means++ guarantee with the Euclidean–Grassmannian comparison when extending the analysis to non-uniform singular values.Source: §C.9 · p. 23

C.10 Proof of Theorem B.6

• The proof splits each function into a fixed W0 component and an adapter-update component, then Rademacher expectation cancels the fixed component.Source: §C.10 · p. 23
• The adapter contribution is rewritten as a Frobenius inner product, enabling a Cauchy–Schwarz bound using rank-one matrices.Source: §C.10 · p. 23

D Additional Experimental Details

• The sensitivity analysis compares K settings on Flan-T5-Base and Flan-T5-Large using GRAM’s clustering partitions and adaptive-rank merging at r=16.Each group is freshly evaluated with the Table 1 pipeline; the caption notes that the Large K=2 row is exactly reproduced and attributes a small Base K=1 difference to SVD tie-breaking.Source: Table 6 · p. 24

D.1 Grouping Analysis

• 4.06 vs. 0.87: GRAM’s K=2 intra-group to inter-group similarity ratio exceeds random grouping’s, and the ratio rises to 4.21 at K=3.Across four grouped settings, higher ratios correlate with better downstream accuracy.Source: Table 7 · p. 24, §D.1 · p. 24

C.10 Proof of Theorem B.6

• The bound is rank-free and applies to any Frobenius-bounded linear adapter; a sharper low-rank bound would require an additional nuclear-norm constraint.The authors do not pursue that sharper bound because the rank-free result suffices to show grouping does not enlarge the hypothesis class.Source: §C.10 · p. 24

D.1 Grouping Analysis

• Random grouping at K=2 yields intra-group similarity of 0.089, below the 0.097 global average, while GRAM avoids this failure mode by construction.Source: §D.1 · p. 24

D.2 Deployment without Oracle Task Identity

• 100.0% of validation examples from every GLUE task route to the correct group at K=2, so task-agnostic GRAM matches oracle GRAM at 83.8 on every task.The Mahalanobis mapper uses five training samples per class (about 85 examples total), requires no gradient training or test-time task labels, and leaves merging fully data-free.Source: §D.2 · pp. 24–25, Table 8 · p. 25
• At every K, the router’s only errors are 72 QQP-to-MRPC confusions among 40,430 QQP examples (0.18%), which are invisible for K ≤5 because GRAM groups those tasks together.They surface at K ≥6 when the tasks are separated; K=8 requires full task identification.Source: §D.2 · p. 25, Table 8 · p. 25

D.3 Per-Task Intrinsic Rank Heterogeneity

• On Flan-T5-Base, the rank needed to capture 99% of Frobenius energy varies across tasks and layers, motivating adaptive rank allocation instead of uniform per-task allocation.Decoder layers typically need 1–4 significant directions, while encoder layers need 10–16; uniform allocation can waste slots on compact spectra or truncate long tails.Source: Figure 4 · p. 25

D.4 Pairwise Projection Error on ViT-B/32

• GRAM’s K=2 partition exposes a block-diagonal projection-error pattern on ViT-B/32, with lower error within groups than across groups.The figure compares K=1 TSV-Merge with K=2 GRAM and reorders tasks by GRAM’s partition.Source: §D.4 · p. 25, Figure 5 · p. 26

D.5 CLIP ViT-B/16 Full Fine-tuning Results

• 91.2% average test accuracy: GRAM is highest among compared methods on the eight-task CLIP ViT-B/16 benchmark.It outperforms Iso-CTS (90.7%), Iso-C (90.0%), and TSV-Merge (88.7%); the largest gains over global merging occur on GTSRB, EuroSAT, and SVHN.Source: Table 9 · p. 26, §D.5 · p. 26

D.6 Implementation Details

• GRAM’s implementation uses K=2 groups, merges matrix weights with SVD truncation, and completes the Flan-T5-Base/GLUE pipeline in under 30 seconds.Vision uses rank 16 per layer, with non-matrix parameters averaged; GRAM and the listed data-free baselines use no calibration data. The appendix also specifies LoRA configurations for GLUE and four Qwen2 medical adapters.Source: §D.6 · pp. 26–27

D.7 Medical Benchmark Details

• The medical benchmark pairs MedQA and CMExam multiple-choice QA with HealthcareMagic and cMedQA2 open-ended dialogue, spanning English and Chinese.MCQA uses exact-match accuracy; the dialogue tasks use BLEU-4 and ROUGE-1, ROUGE-2, and ROUGE-L (F1), yielding 10 metric columns.Source: §D.7 · p. 27
• Medical benchmark Avg Rank is the mean rank across 10 columns, while Avg Impr averages each method’s per-column relative improvement over the un-adapted Pretrained baseline.Rank 1 is best, and the per-task Individual oracle is excluded from ranking; relative improvement normalizes across metrics with different scales.Source: §D.7 · p. 27