PaperlayerView exact revision on arXiv

Source-linked AI summary

Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

Minsik Choi, Geewook Kim

arXiv:2606.01717v1cs.LG

TL;DR

Heterogeneous instruction mixtures face conflicting gradients and costly synchronization. MERIT splits datasets along PCA conflict axes, trains branches independently, and merges them once, improving the 8-benchmark average from 54.3 to 57.0 on Qwen2.5-VL-3B.

  • Problem

    Centralized joint training of heterogeneous mixtures is limited by conflicting gradients and bandwidth-heavy synchronization across distributed hardware.

  • Method

    MERIT uses PCA-based gradient-conflict splitting, independent branch fine-tuning without cross-partition communication, and one token-weighted parameter merge.

  • Results

    54.3 →57.0: MERIT improves the 8-benchmark average over joint training on Qwen2.5-VL-3B across 136 Vision-FLAN tasks, while matching or exceeding joint training at 7B scale.

  • Takeaways & Limitations

    MERIT supports decentralized instruction tuning for multimodal and text-only mixtures with consistent empirical gains and minimal cost overhead at the reported scales.

  • Takeaways & Limitations

    MERIT requires a merge-ready initialization whose independently fine-tuned models remain in a connected low-loss region, an empirical property requiring diagnostics.

Abstract

from arXiv · show

Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture -- matching or exceeding centralized joint training with minimal cost overhead -- and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.

1. Introduction

MERIT addresses the coupled optimization and communication bottlenecks of heterogeneous instruction tuning by splitting tasks along PCA-identified gradient conflicts, training partitions independently, and merging them once in parameter space. A local quadratic theory explains this design through curvature-weighted variance reduction, PCA alignment, implicit regularization, and spectral filtering, with empirical gains across multimodal and text-only settings.

  • Motivation: Heterogeneous mixtures cause gradient interference, negative transfer, stiff dynamics, and bandwidth-heavy synchronization that limit centralized joint training.These constraints are especially problematic in fragmented compute environments where communication is costly or unavailable.
  • Method: MERIT partitions the mixture along top-r PCA conflict axes, trains K=2r branches independently, and merges them once via token-weighted averaging.Conflicts are estimated from a small calibration set at a merge-ready initialization, with no cross-partition gradient communication during fine-tuning.
  • Theory: Weight averaging also provides implicit norm regularization toward θ(0) and conceptually acts as spectral filtering.These effects are part of the theoretical justification for reconciling independently trained branches in parameter space.
  • Theory: The local quadratic theory shows that merging yields curvature-weighted variance reduction, while PCA-aligned splitting maximizes the gain and benefits more from a larger Hessian spectral gap.The analysis assumes fine-tuning remains within a shared flat basin.
  • Results: 54.3 →57.0: MERIT improves the 8-benchmark average on Qwen2.5-VL-3B with 136 Vision-FLAN tasks under identical token budgets.On Qwen2.5-VL-7B with a 1.6M-example, 176-source mixture, it matches or exceeds centralized joint training across three seeds with minimal wall-clock overhead and transfers to text-only FLAN.

2. Background and Related Work

MERIT connects model merging, decentralized optimization, and heterogeneous instruction-mixture learning. It differs from prior approaches by using an a priori conflict-aware dataset split, communication-free local training, and one-shot merging under a flat-basin theory.

  • Model Merging and Loss-Landscape Connectivity: Model soups and Model Stock show that averaging independently fine-tuned checkpoints can be effective, while prior work mainly merges models trained on the same dataset or objective.These settings primarily reduce run-to-run variance; the paper therefore includes soup-style and random-partition averaging baselines.
  • Federated and Decentralized Optimization: FedAvg and Local SGD periodically synchronize client models, whereas one-shot federated learning removes iterative communication but does not provide MERIT’s conflict-aware partitioning.Federated LLM fine-tuning also uses compression, seed-based tuning, perturbation, low-rank, and personalization methods to reduce per-round costs.
  • Task Interference in Multi-Task and Instruction Tuning: Gradient reweighting and projection mitigate negative transfer but typically require synchronized per-task gradients, while instruction tuning remains sensitive to mixture composition.This motivates moving conflict handling before training rather than relying on synchronized gradient access.
  • Data Curation and Mixture Design for (M)LLMs: MERIT complements iterative mixture curation with a reusable decomposition primitive that estimates dataset interactions at a shared initialization and enables communication-free parallel fine-tuning.The approach addresses heterogeneous datasets whose ratios and priorities may change over time.
  • Positioning of Our Work: MERIT introduces an a priori conflict-aware dataset split so simple averaging works with disjoint heterogeneous data, unlike post-hoc merging and centralized-gradient methods.Its analysis assumes a flat-basin merging regime rather than treating federated averaging as an approximation to centralized optimization of one objective.

3. Theoretical Framework

The theoretical framework shows that, within a shared flat basin, weight merging reduces curvature-weighted variance, while PCA-based conflict splitting aligns dispersion with high-curvature disagreement directions. It further interprets merging as spectral filtering and implicit norm regularization, motivating MERIT’s merge-ready design.

  • Merging gain: Under a local quadratic model, merging never increases weighted loss and strictly improves it when checkpoint displacements differ outside H’s nullspace.The gain is governed by curvature-weighted variance and is maximized when dispersion concentrates along dominant Hessian eigendirections.
  • Conflict-aware splitting: PCA-aligned partitioning maximizes merging gain by selecting high-curvature disagreement axes, outperforming random partitioning under the stated linear-quadratic conditions.It achieves the maximum among balanced partitions in the T=4, d=2 instance and exceeds random partitioning in expectation under a spectral-concentration condition.
  • Conflict-aware splitting: MERIT recursively splits along top-r PCA axes, accumulating gains across r orthogonal high-curvature directions.The framework connects gradient conflicts to curvature through gt ∝ H∆t, while gradient disagreement provides a first-order proxy for curvature-sensitive update directions.
  • Spectral filtering: PCA-structured splitting and merging suppress dominant-curvature error components, acting as approximate spectral filtering when joint-training stability is limited by λmax.Joint training requires η < 2/λmax, which can slow progress along flatter dimensions when the condition number κ is large.
  • Implicit norm regularization: Weight averaging contracts the merged model toward the shared initialization, reducing a distance-based complexity term in PAC-Bayes bounds.This provides a theoretical account of improved generalization even when individual branch training losses are not lower than joint training.

4. Proposed Method

MERIT is a decentralized instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions heterogeneous mixtures along dominant PCA axes, trains partitions independently, and merges checkpoints once by token-weighted averaging. It requires a merge-ready initialization whose independently fine-tuned models remain in a connected low-loss region.

  • Pipeline overview: MERIT creates K=2r conflict-aware partitions, fine-tunes each from shared initialization θ(0), and merges checkpoints through token-weighted averaging.The five stages are conflict estimation, PCA decomposition, balanced partitioning, communication-free branch training, and weight merging.
  • Gradient conflict estimation: MERIT estimates representative per-dataset gradients from calibration examples, normalizes them, and constructs a cosine-similarity matrix to quantify interference.Gradients use identical model and trainable-parameter settings; up to 200 examples per dataset are used for calibration.
  • PCA conflict decomposition: PCA extracts an r-dimensional conflict embedding z_t for each dataset, with r ∈ {1, 2, 3} producing K=2^r branches.Cosine-similarity PCA is scale-invariant and recovers the leading eigenspace of raw-gradient PCA under gradient-norm concentration.
  • Conflict-aware partitioning: Datasets with similar PCA projections share groups, while opposing projections are separated to distribute updates across distinct conflicting directions.Recursive sample-balanced median splits approximate 50/50 group sizes without sacrificing separation along dominant conflict axes.
  • Communication-free training: Each group is fine-tuned independently from θ(0) with shared settings, full per-dataset budgets, and no cross-group communication.Branches can run in parallel on disjoint hardware, while total training budget matches centralized joint training.
  • Parameter-space merging: The branch checkpoints are merged in one pass by token-weighted averaging, which becomes uniform only when per-group token budgets are exactly balanced.Sample-balanced splitting makes budgets approximately, rather than exactly, balanced, so merging is generally non-uniform.

5. Experiments

Experiments evaluate MERIT on controlled multimodal instruction-tuning studies spanning 3B and 7B settings. MERIT consistently improves over joint training and random splitting, with gains attributable to conflict-aware PCA-based partitioning.

  • Setup: The controlled 3B study uses Qwen2.5-VL-3B, 136 Vision-FLAN tasks, and eight multimodal benchmarks across four categories.Baselines include centralized joint training, random partitioning, conflict-induced splitting, and uniform model soups.
  • Results: Every MERIT variant outperforms single-epoch joint training on the 8-benchmark average, including random partitioning.This pattern is consistent with a deterministic merging gain within a shared flat basin.
  • Results: +2.7 is the improvement over the 1 ep baseline without any cross-partition gradient communication during fine-tuning.The pattern replicates at 7B scale across three independent seeds.
  • Conflict-aware splitting: +2.5 is MERIT-3D’s improvement over Random (8 groups) under identical budgets and the same one-shot merging step.The difference is attributable purely to the choice of split.
  • Conflict-aware splitting: The advantage grows monotonically with the number of PCA dimensions, whereas random partitioning shows no such trend.MERIT also outperforms K-means on the same gradient representations, indicating that conflict-direction alignment matters more than grouping similar datasets.

6. Further Analyses and Discussions

MERIT’s conflict-aware split-and-merge recipe generalizes to 7B-scale multimodal instruction tuning, stronger initializations, and text-only FLAN. It also preserves open-ended quality, incurs modest recurring overhead, and outperforms alternative post-hoc merging operators.

  • Scaling and stronger initializations: 60.9 →61.5: MERIT again exceeds Joint FFT on the overall average from a stronger initialization using 2.9M image-captioning samples.The headroom over the base model is smaller for both methods.
  • Scaling and stronger initializations: MERIT preserves open-ended quality while Joint FFT collapses LLaVA-Wild from 67.1 →50.2, with MERIT scoring 66.2.In the primary 3B comparison, Joint training drops LLaVA-Wild from 53.2 to 41.9 while MERIT preserves it at 52.0.
  • Text-only transfer: +0.8: MERIT-2D achieves the best average on 66 text-only FLAN tasks, exceeding 1-epoch joint training and matching or slightly exceeding 2-epoch joint training at half the budget.This extends conflict-aware splitting beyond multimodal settings.
  • Decentralized efficiency: ∼24%: MERIT’s 3B parallel training and one-shot merge exceed single-epoch joint training on 8 V100 GPUs, while 7B recurring overhead falls to 0.8% on 8 A100 GPUs.Dataset-level gradient-conflict preprocessing is one-time and amortizable, taking ∼2h at 3B scale.
  • Post-hoc merging: 5–15 points: token-weighted averaging consistently outperforms TIES, STAR, TSV, and Iso-CTS as drop-in mergers on MERIT’s 7B 2D branches.The comparison averages results over three seeds on the 8-benchmark suite.

7. Conclusion … A.1.4. IMPLEMENTATION DETAILS

MERIT combines conflict-aware dataset splitting, independent branch training, and one-time token-weighted merging, supported by a local quadratic analysis of variance reduction, PCA alignment, spectral filtering, and implicit norm regularization. The appendix specifies shared datasets, benchmarks, baselines, and Qwen model training configurations for multimodal and text-only experiments.

  • 7. Conclusion: MERIT uses gradient-conflict PCA to split datasets, trains group-wise branches without cross-group synchronization, and merges them once via token-weighted averaging.Its analysis links merging to curvature-weighted variance reduction, PCA-aligned conflict splitting, spectral filtering, and implicit norm regularization.
  • A.1. Experimental Setup: The experimental setup described in Appendix A.1 is shared across experiments unless otherwise specified.
  • A.1.1. TRAINING DATASETS: Vision-language experiments use 136 of 187 Vision-FLAN tasks at 3B, while text-only experiments use 66 FLAN instruction tasks with shared mixtures across methods.All comparison methods share the same Vision-FLAN splits and training data, and all FLAN baselines and MERIT variants use the same task mixture.
  • A.1.2. EVALUATION BENCHMARKS: Multimodal headline comparisons use eight benchmarks spanning reasoning, open-ended understanding, text-centric perception, and expert multimodal reasoning.The clustering ablation uses an 11-benchmark protocol adding MME, HallusionBench, DocVQA, and MIABench, while excluding LLaVA-Wild.
  • A.1.2. EVALUATION BENCHMARKS: Text-only evaluation covers MMLU, HellaSwag, WinoGrande, ARC-C, HumanEval, BoolQ, GPQA, and XNLI under standard evaluation protocols.
  • A.1.3. BASELINES: Joint training treats each setting’s datasets as one corpus and reports the best configuration across 0.5, 1, and 2 epochs after per-device batch-size grid search.MERIT branches use one fixed batch size, so reported gaps understate MERIT’s advantage under matched tuning budgets.
  • A.1.3. BASELINES: Baselines include random splits into 2, 4, or 8 groups, uniform soups of 2, 3, or 4 models, and a greedy conflict-induced split, all merged by weight averaging.Random-split groups train independently; uniform soups train on the full dataset with different data orders; conflict-induced groups minimize within-group average cosine similarity.
  • A.1.4. IMPLEMENTATION DETAILS: Text-only experiments use Qwen2.5-3B, while vision-language experiments use Qwen2.5-VL-3B-Instruct with the vision encoder and multimodal projector frozen.Only the language decoder is updated for vision-language fine-tuning, images use a maximum resolution of 784×784 pixels, and methods share initialization and training configuration.

A.2. Large-Scale Vision–Language Model Experiments · B. Merge-Readiness Diagnostics · B.1. Linear Mode Connectivity

The paper details a 7B merge-ready vision–language setup built from Qwen2.5 components and a 1.6M-example, 176-unit mixture, then evaluates merge-readiness through complementary diagnostics including interpolation barriers.

  • A.2. Large-Scale Vision–Language Model Experiments: The 7B study builds a vision–language base by pairing the Qwen2.5-VL vision encoder with Qwen2.5-7B-Instruct and training it with a LLaVA-style recipe.The study does not start from a released vision–language checkpoint.
  • A.2. Large-Scale Vision–Language Model Experiments: The Base VLM uses feature alignment followed by a 0.7M-example instruction-tuning stage, while the Scaled base VLM adds a 2.9M-example re-captioned corpus beforehand.The Scaled base VLM therefore uses feature alignment, knowledge learning, and instruction tuning.
  • A.2. Large-Scale Vision–Language Model Experiments: The large-scale FFT mixture contains 1.6M examples assembled from publicly available datasets and converted into a unified multimodal instruction format.Examples use image, instruction, and response fields compatible with the LLaVA-style pipeline.
  • A.2. Large-Scale Vision–Language Model Experiments: The Vision-FLAN and The Cauldron sources use curated upstream task subsets, with the exact list of 176 task-unit identifiers and scripts slated for release.The datasets are publicly available under their original licenses.
  • A.2. Large-Scale Vision–Language Model Experiments: Each task-specific subset is preserved as a dataset unit, yielding 176 units for gradient-conflict estimation and MERIT partitioning.The same 1.6M FFT mixture and training budget are used for Joint FFT and MERIT.
  • B. Merge-Readiness Diagnostics: Merge-readiness diagnostics collectively verify the initialization assumption underlying MERIT.The section consolidates four complementary diagnostics.
  • B.1. Linear Mode Connectivity: The linear-mode-connectivity test evaluates loss barriers on all pairwise and branch-to-merged paths in the 3B 2D split with K=4.It uses 21 evenly spaced interpolation points over α ∈[0, 1] at step 0.05, defining the barrier as the maximum excess above linear interpolation.

B.2. Weight Perturbation Robustness … C.2. Robustness and Multi-Seed Reproducibility

The analyses show that merging yields a flatter, more regularized solution while contracting parameters toward initialization, and that gradient structure supports conflict-aware partitioning. Robustness evaluation further examines reproducibility across independent training seeds at both model scales.

  • B.2. Weight Perturbation Robustness: Merged models show consistently lower loss sensitivity and flatness AUC than joint training under Gaussian perturbations across σ ∈ {0.01, 0.05, 0.1} and epochs 1–3.This provides direct evidence that merging places the solution in a flatter loss-landscape region.
  • B.2. Weight Perturbation Robustness: All 10 linear-mode-connectivity barriers are exactly 0 for the 2D split with K=4, confirming that branches remain in a shared flat basin.The result indicates no loss barrier between the independently trained branches.
  • B.3. Displacement Contraction: The merged model remains 2–3× closer to the shared initialization throughout training, with the ratio widening monotonically.This empirically confirms the convexity-based displacement contraction from weight averaging.
  • B.4. Training Loss vs. Generalization: Merged models have substantially higher training loss than joint training yet better held-out performance, with the gap widening monotonically over training.This pattern is identified as the classic signature of implicit regularization from merging.
  • C.1. Visualization of Dataset-Level Gradients: Dataset-level gradients cluster by task type in a two-dimensional t-SNE projection, with VQA, image classification, and captioning occupying distinct regions.The visualization uses gradients at θ(0) from 136 Vision-FLAN tasks on Qwen2.5-VL-3B and overlays KDE contours.
  • C.1. Visualization of Dataset-Level Gradients: MERIT exploits separable regions of disagreeing gradients by partitioning datasets along PCA conflict axes to group compatible datasets together.This connects the observed task-type structure in gradient space to the conflict-aware split.
  • C.2. Robustness and Multi-Seed Reproducibility: Reproducibility is assessed over five independent seeds for the 3B primary comparison and three independent seeds for the 7B replication.The 3B evaluation uses a paired Wilcoxon test; alternative post-hoc merging operators are evaluated separately in Appendix C.5.
  • C. Additional Analyses and Ablations: Appendix C.5 isolates aggregation-rule choice from training-seed variability by comparing alternative post-hoc merging operators on the same 7B 2D branches.These per-seed comparisons address a different question from the primary multi-seed reproducibility analysis.

C.2.1. 3B FIVE-SEED COMPARISON … D. Additional Qualitative Analysis

Across repeated evaluations, MERIT consistently outperforms joint training at 3B and 7B, while PCA-based partitioning remains stronger than K-means and random splitting. Calibration analysis supports using 200 samples per dataset, and the supplied passages describe post-hoc merging baselines without reporting their numerical outcomes.

  • D. Additional Qualitative Analysis: The supplied passages contain no substantive qualitative-analysis finding for section D.No evidence passage labeled D. Additional Qualitative Analysis is provided in the input.
  • C.2.1. 3B FIVE-SEED COMPARISON: MERIT outperforms Joint training on the 8-benchmark average in all five independent 3B runs.The comparison uses CosSim PCA, a 3D split, 8 groups, and one joint-training epoch across four benchmark categories.
  • C.2.1. 3B FIVE-SEED COMPARISON: p = 0.03125, a one-sided paired Wilcoxon signed-rank test result significant at the 5% level.The test uses five paired per-run averaged scores.
  • C.2.2. 7B THREE-SEED REPLICATION: MERIT outperforms joint training in all three independent 7B training-seed runs.The replication uses Qwen2.5-VL-7B on 176 tasks; open-ended reasoning improves while factual benchmarks remain within noise.
  • C.3. Clustering Strategies for Dataset Partitioning: PCA-based MERIT consistently outperforms K-means and scales monotonically with dimensionality.K-means plateaus at a lower aggregate and shows non-monotonic patterns on MathVista, HallusionBench, and MMVet.
  • C.3. Clustering Strategies for Dataset Partitioning: Random partitioning can outperform joint training, but PCA splitting concentrates dispersion along high-curvature conflict axes for larger Hessian-weighted gains.Random splits benefit from variance cancellation but do not control covariance alignment with curvature.
  • C.4. Calibration Dataset Size Sensitivity: 0.847 mean cosine similarity is achieved at n=200 calibration samples per dataset against the full 1,000-sample reference gradient.Across 100 qualifying Vision-FLAN tasks, the standard deviation is 0.106 and returns diminish beyond 200 samples.
  • C.5. Post-Hoc Merging Baselines: Post-hoc merging baselines replace token-weighted averaging on MERIT’s 7B 2D branches with K=4 across three independent seeds.Each baseline uses its official repository’s default hyperparameters; the supplied passage does not report numerical results.

D.1. Short-Answer Collapse on LLaVA-Wild · D.2. Qualitative Examples on Multimodal Reasoning

Joint training suffers short-answer collapse on LLaVA-Wild, while MERIT preserves more open-ended behavior by separating conflicting subsets before fine-tuning. Qualitative examples show MERIT generally produces more informative, context-aware multimodal answers, especially as decomposition dimensionality increases.

  • D.1. Short-Answer Collapse on LLaVA-Wild: 53.2 →41.9–42.8: Joint training degrades LLaVA-Wild in the 3B setting through short-answer collapse under heterogeneous mixtures.The model produces overly terse responses.
  • D.1. Short-Answer Collapse on LLaVA-Wild: 67.1 to 50.2: Joint FFT on the Scaled base collapses LLaVA-Wild at 7B, whereas MERIT preserves the base model’s open-ended quality at 66.2.MERIT mitigates collapse by separating conflicting subsets before fine-tuning.
  • D.1. Short-Answer Collapse on LLaVA-Wild: In a joke explanation, Joint gives a detailed interpretation, whereas MERIT outputs range from a generic cartoon explanation to a substantially more elaborate computer-science interpretation.The 3D response explains the cartoon’s simple two-layer stacking example and why that makes the joke funny.
  • D.2. Qualitative Examples on Multimodal Reasoning: Joint-trained models tend to give terse or generic responses, while MERIT models provide more informative and context-aware answers.Representative LLaVA-Wild examples include joke explanation and sketch description prompts.
  • D.2. Qualitative Examples on Multimodal Reasoning: Higher decomposition dimensionality often, though not uniformly, yields richer and more context-aware answers.The largest gains occur for responses requiring multi-step reasoning or named-entity recall, including Examples 1 and 2.

E. Detailed Efficiency Analysis · E.1. Comparison with Centralized Gradient Methods · F. Theoretical Analysis

MERIT achieves favorable runtime while avoiding the memory and coordination costs of centralized per-step conflict-resolution methods. Its efficiency analysis also shows reusable similarity computation, high-fidelity gradient sampling, and formal theoretical support for the paper’s three stated implications.

  • E. Detailed Efficiency Analysis: ∼1.6h and ∼2.0h: gradient extraction dominates preprocessing for 136 and 176 datasets, while cosine computation takes ∼28–38 min and PCA takes < 1s.Uniform gradient sampling preserves cosine similarity structure with high fidelity, with ρ>0.98.
  • E. Detailed Efficiency Analysis: O(Tm): adding m datasets to an existing T-dataset collection requires only crosssimilarity computations plus a negligible PCA update, rather than full O((T+m)2) recomputation.The similarity matrix is therefore reusable as the dataset collection grows.
  • E. Detailed Efficiency Analysis: 5h24m: MERIT-3D is faster than 2-epoch joint training at 8h40m and only modestly slower than 1-epoch joint training at 4h22m on V100.At 7B scale, MERIT adds 21 minutes (0.8%) over 1-epoch joint training, 43h39m versus 43h18m on A100.
  • E.1. Comparison with Centralized Gradient Methods: 136 tasks: centralized per-step conflict-resolution feasibility is analyzed at the experimental scale, where pairwise projections and repeated gradient processing create substantial resource demands.The supplied comparison passage identifies PCGrad as requiring T backward passes +.
  • E.1. Comparison with Centralized Gradient Methods: ∼816 GB: storing 136 full gradients exceeds the V100×8 capacity of 256 GB and A100×8 capacity of 640 GB, with a conservative estimate of >17 days per epoch.The analysis considers T=136 tasks and a 3B-parameter model; GradNorm also requires centralized coordination from all 136 tasks at every step.
  • F. Theoretical Analysis: F. Theoretical Analysis: the appendix provides formal assumptions, proofs, and extended analyses supporting the three implications stated in Section 3.These materials collect the formal backing for the paper’s theoretical claims.

F.1. Quadratic Analysis of Merging in Flat PCA-Structured Basins … F.5. Implicit Regularization via Averaging-Induced Contraction

The appendix develops a quadratic account of merging, showing that curvature-weighted variance reduction is amplified by PCA-aligned conflict splitting and accompanied by optimization and norm-regularization benefits. It further justifies cosine-similarity PCA as a scale-invariant proxy for gradient structure and empirically confirms its partial alignment with high-curvature Hessian directions.

  • F.1; STEP 1: QUADRATIC AVERAGING YIELDS A DETERMINISTIC GAIN: Merging never hurts under the quadratic model, with gains equal to checkpoint variance measured in the Hessian geometry.The gain is largest when disagreement lies in sharper directions and is strict whenever checkpoint differences project onto range(H).
  • STEP 2: ACCOUNTING FOR DATASET-SPLIT MISMATCH; STEP 3: EXTENSION TO THE TRUE POPULATION LOSS: Under smooth split mismatch, the true-loss benefit is governed by competing curvature spectra, with PCA placing dispersion where merging gain can dominate residual disagreement.The population-loss decomposition contains exact variance reduction plus cubic Taylor and second-order mismatch remainders; small displacements make the cubic term negligible.
  • STEP 4: PCA-STRUCTURED PARTITIONING: PCA-based partitioning enlarges displacement variance along dominant curvature directions, and in the ideal symmetric case cancels the high-curvature loss component after merging.The gradient-defined and Hessian-curvature subspaces need only be strongly coupled, not identical.
  • STEP 5: OPTIMIZATION ADVANTAGE OVER JOINT TRAINING (CONCEPTUAL): Merging suppresses sharp-subspace error in one step, replacing stability-limited joint descent with immediate cancellation of the stiffest conflicts.Joint training must use η < 2/λmax for stability, slowing progress in flat directions when conflicting gradients fluctuate along sharp ones.
  • F.2; F.2.2. COSINE SIMILARITY AS A NORMALIZED GRAM MATRIX; F.2.3. SUBSPACE EQUIVALENCE UNDER GRADIENT NORM CONCENTRATION; F.2.4. DIRECTIONAL INTERPRETATION VIA NORMALIZED GRADIENTS: Cosine-similarity PCA removes per-dataset gradient magnitudes while preserving directional alignment, recovering the leading raw-gradient subspace when gradient norms are sufficiently concentrated.Across 136 Vision-FLAN datasets, trimming reduces coefficient of variation from 0.39 raw to 0.23 at 5% and 0.17 at 10%.
  • F.2.1. PROBLEM SETUP AND NOTATION; F.2.5. EMPIRICAL AND PRACTICAL IMPLICATIONS: At 1D and 2D, raw-gradient and cosine-similarity PCA produce nearly identical averages within 0.2 points and the same ordering, while their 3D behavior diverges in favor of cosine similarity.This empirical result motivates the scale-invariant cosine construction used by MERIT.
  • F.3. Empirical Hessian–PCA Alignment; F.4. Formal Justification of Gradient–Curvature Alignment in a Tractable Setting: Gradient-PCA conflict directions partially align with top Hessian eigenvectors, with leading principal angles of 55–70° and Gaussian-baseline z-scores ≥104 across both tested model scales.In the tractable linear-quadratic model, gradient covariance equals HΣθH, so high-curvature and high-disagreement directions are amplified and PCA-aligned splitting maximizes merging gain under the stated conditions.
Loading 2606.01717v1…