PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Yuanyi Wang, Yifan Yang, Su Lu, Yanggan Gu, Pengkai Wang, Wenjun Wang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Jialun Cao, Shing-Chi Cheung, Hongxia Yang

arXiv:2605.09608v1cs.LGcs.IT

TL;DR

Continual post-training lacks clear criteria for when sequential updates transfer capabilities or cause catastrophic forgetting. This paper analyzes task-induced covariance geometry and introduces GCWM, which uses geometry conflict to control data-free update integration. Across domain- and capability-continual settings, GCWM consistently improves retention and final performance over data-free baselines across Qwen3 scales.

  • Problem

    It remains unclear when sequential post-training updates transfer capabilities or interfere, limiting criteria for deciding whether new updates should be integrated.

    Source: §1 · p. 2

  • Method

    The paper represents tasks by parameter updates, measures state-relative covariance geometry conflict, and uses it to gate geometry-aware correction in data-free GCWM.

    Source: §1 · p. 2

  • Results

    GCWM consistently improves retention and final performance over data-free baselines across domain- and capability-continual settings on Qwen3 models from 0.6B to 14B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Geometry conflict serves as both an explanatory signal for forgetting and a practical control signal for sequential update integration.

    Source: §6 · p. 10

    The evidence focuses on Qwen3-scale open LLMs and public domain- and capability-continual benchmarks, while GCWM requires task updates and adds geometry-computation cost.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.

1 Introduction

This introduction frames LLM continual post-training as a compatibility problem: sequential updates forget when their covariance geometries conflict with the evolving model state, but transfer when they remain compatible. It proposes GCWM, a data-free, geometry-conflict-gated Wasserstein merging method that improves retention and final performance across Qwen3 scales without replay data.

Motivation

  • Existing sequential fine-tuning, replay, regularization, and model merging methods lack a principled criterion for deciding when new updates should be integrated strongly or restrained.This challenge is amplified by heterogeneous LLM tasks and differing post-training objectives, where equal update magnitudes can yield different retention outcomes.Source: §1 · pp. 1–2

Task geometry

  • Geometry conflict is defined as a normalized Bures–Wasserstein discrepancy between task-induced covariance geometries in a shared space, with a state-relative form measuring compatibility.The representation captures update magnitude, affected subspaces, and spectral structure.Source: §1 · p. 2
  • Forgetting is a state-relative update-integration failure caused by misalignment between task-induced covariance geometries and the evolving model state, while compatible updates transfer.This geometry-based account explains why raw update norm and isolated pairwise compatibility are insufficient.Source: §1 · p. 2

Method

  • GCWM constructs task-induced covariance geometry, builds a shared Wasserstein metric using Gaussian Wasserstein barycenters, and gates geometry-aware correction by conflict.The method provides compatibility-controlled update integration, with theory linking loss change to geometry conflict and gated merge displacement.Source: §1 · p. 2

Empirical scope

  • GCWM consistently improves retention and final performance over data-free baselines across domain-continual and capability-continual settings on Qwen3 models from 0.6B to 14B, without replay data.The method remains the strongest data-free update-integration approach across the evaluated scales.Source: §1 · p. 2

2 Preliminary

The paper models continual post-training as sequential task-specific parameter updates and analyzes compatibility through the covariance geometry those updates induce. It positions geometry conflict as a task-compatibility signal for understanding and controlling sequential interactions.

2.1 Problem Setup

  • Continual post-training adapts a pretrained LLM through sequential tasks introducing new domains, skills, or behaviors, using parameter-efficient or full-model updates as analysis objects.For task T_t, the update is defined relative to the model adapting to that task.Source: §2.1 · pp. 2–3

2.2 Task Geometry and Compatibility Signals

  • Task geometry captures dominant update directions, while normalized Bures–Wasserstein discrepancy measures geometry conflict between tasks; lower values indicate more compatible geometries.State-relative variants replace one task update with the current continual-training state, and comparisons include update norm, SAR [24], and gradient cosine conflict [25].Source: §2.2 · p. 3

2.3 Related Work

  • Existing continual-learning and merging methods address forgetting through sequential fine-tuning, replay, regularization, or model composition, but provide limited guidance on when task interactions should be encouraged or suppressed.This work instead introduces geometry conflict as a method-native control signal and builds a shared merging metric using Bures–Wasserstein geometry and Gaussian Wasserstein barycenters [52].Source: §2.3 · p. 3

3 What Governs Forgetting in Continual Post-Training?

Forgetting is best explained by state-relative geometry mismatch rather than update magnitude or isolated pairwise compatibility. Geometry conflict and gradient conflict expose complementary failure modes, motivating geometry conflict as a control signal for update integration.

3.1 Update Norm Is Insufficient to Explain Forgetting

  • Update norm measures model movement but not compatibility, showing only a coarse association with retention loss across Qwen3 models and continual-training strategies.The analysis compares Seq. SFT, EWC, FOREVER replay, and AIMMerging across Qwen3 models from 0.6B to 14B.Source: §3 · pp. 3–4, §3.1 · p. 4
  • State-relative geometry best tracks forgetting, with global state-active correlation |ρs| = 0.59 versus 0.48 for update norm and 0.30 for active-pair conflict.Across scales, the global gap rises from 0.16 at 0.6B to 0.86 at 14B, while update norm remains a weaker drift baseline.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Geometry Conflict Refines Subspace Compatibility

  • Pairwise geometry conflict refines SAR but is non-redundant and insufficient alone: their global rank association is ρs = 0.27, while pairwise forgetting correlations vary by scale.Geometry conflict captures covariance compatibility in shared directions, whereas SAR captures overlap; large drops such as Math→History (12.9 pp) lack one pairwise-conflict pattern.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 State-Relative Geometry Conflict Tracks Continual Forgetting

  • State and global geometry gaps follow retention-loss growth more closely than active-pair conflict, especially from 1.7B to 14B and under sequential updating.Their correlations reach 0.68/0.70 for Seq. SFT and 0.40/0.42 for EWC, weakening when replay or merging compresses forgetting variance.Source: §3.3 · p. 5

3.4 Geometry and Gradient Conflict Reveal Complementary Failure Modes

  • Geometry conflict and gradient conflict are complementary: geometry concentrates in up_proj, gate_proj, v_proj, and down_proj, while negative-gradient conflict concentrates in k_proj and q_proj.The four geometry-heavy families account for about 0.89 of top geometry-conflict layers, versus about 0.86 for query/key projections among top negative-gradient layers.Source: §3.4 · p. 5

4 Geometry Conflict Wasserstein Merging

GCWM is a data-free, compatibility-controlled merging method that represents task updates through covariance geometry, gates geometry-aware correction by conflict, and applies only incremental merged changes. Its theory bounds the correction’s relative loss effect using geometry conflict and metric displacement, with stronger correction assigned to higher-conflict layers.

4 Geometry Conflict Wasserstein Merging

  • GCWM integrates task vectors by estimating layer-wise geometry conflict, constructing a shared Wasserstein metric, and gating geometry-aware correction during continual post-training.The method converts state-relative geometry into an actionable control signal for update integration.Source: §4 · p. 6, §4.1 · p. 6

4.1 Task Geometry and Conflict Gate

  • GCWM represents active updates with layer-wise covariance geometries and uses normalized Bures–Wasserstein discrepancies to compute pairwise conflict and layer-wise gates.The gates depend on normalized task-pair weights, a conflict threshold, and gate sharpness.Source: §4.1 · p. 6

4.2 Shared Wasserstein Metric and Gated Merge

  • A Gaussian Wasserstein barycenter defines a shared local metric, where GCWM whitens projected updates, applies weighted WUDI [55], recolors the result, and blends it with a plain merge.The implementation uses a regularized full-space transform corresponding to the projected formulation.Source: §4.2 · pp. 6–7

4.3 Incremental Continual Update

  • At each continual step, GCWM selects an active set containing the current update and optional historical information, then applies only the change from the previous merged state.This incremental differencing keeps post-training tied to newly induced compatibility-controlled changes.Source: §4.3 · p. 7

4.4 Theoretical Support for GCWM

  • Theorem 1 bounds GCWM’s additional loss relative to plain merging by shared geometry conflict and metric displacement, while Proposition 1 assigns weaker correction to low-conflict layers and stronger correction to high-conflict layers.Together, the results control geometry-aware integration at both the loss and update levels under the stated assumptions.Source: §4.4 · p. 7

5 Experiments

Experiments evaluate GCWM as a data-free update-integration method across domain-continual and capability-continual settings. GCWM achieves the strongest reported non-MTL or data-free performance while improving over competing data-free baselines across model scales and benchmarks.

5.2 Domain-Continual Post-Training

  • GCWM gives the strongest non-MTL domain-continual performance at 1.7B, 8B, and 14B, improving over the best data-free baseline by +1.61, +0.74, and +1.23 points.The gains span 12/14, 10/14, and 9/14 domains at the respective scales when compared with AIMMerging.Source: Table 1 · p. 8, §5.2 · pp. 8–9

5.1 Setup

  • The evaluation uses Qwen3 models from 0.6B to 14B, 14 MMLU-Pro domain tasks, and six capability benchmarks, averaging scores over five runs.Main comparisons target data-free methods, while sequential SFT, EWC, and FOREVER serve as continual-training references using regularization or replay.Source: §5 · p. 7, §5.1 · p. 8

5.3 Capability-Continual Post-Training

  • GCWM achieves the best data-free average at 1.7B (62.6 vs. 56.8 for OPCM, +5.78 points) and 14B (74.3 vs. 72.9).At 1.7B it leads on all six benchmarks; at 14B it leads on GPQA-Diamond, GSM8K, HumanEval, and MMLU-Pro.Source: Table 2 · p. 9, §5.3 · p. 9

6 Conclusion

The conclusion identifies forgetting as a state-relative update-integration failure caused by incompatibility between task-induced covariance geometries and the evolving model state. It presents GCWM as a geometry-conflict-gated, data-free integration method that improves retention and final performance without replay data.

Conclusion

  • Forgetting arises when task-induced covariance geometries become incompatible with the evolving model state, making geometry conflict an explanatory signal for harmful updates.This state-relative view explains why raw drift and isolated pairwise compatibility are insufficient.Source: §6 · p. 10
  • GCWM constructs a shared Wasserstein metric from task-induced covariance geometry and gates data-free update integration using geometry conflict.The method operationalizes geometry conflict as a control signal for sequential update integration.Source: §6 · p. 10
  • Across domain-continual and capability-continual settings, GCWM improves retention and final performance over data-free baselines without replay data.Source: §6 · p. 10

Limitations · A Discussion and Broader impacts

The study’s evidence is limited to Qwen3-scale open LLMs and selected continual post-training benchmarks, while GCWM is intended as an explanatory and control signal rather than a universal causal account of forgetting. GCWM reduces reliance on replay but requires task updates, offline computation, and standard safety governance because easier adaptation can enable harmful behaviors.

Limitations

  • GCWM’s main evidence comes from Qwen3-scale open LLMs and domain- and capability-continual tasks covering reasoning, knowledge, math, and code benchmarks.Source: Limitations · p. 16
  • The state-relative geometry signal is consistent across tested scales and methods but does not prove causal necessity for every form of forgetting.Source: Limitations · p. 16
  • GCWM is most relevant when historical data are unavailable or replay is undesirable, whereas replay-based training can remain stronger when repeated access to past data is permitted.Source: Limitations · p. 16
  • The method assumes task-specific updates or checkpoints and adds offline CPU cost for geometry construction and Wasserstein metric computation, without affecting inference.Source: Limitations · p. 16

A Discussion and Broader impacts

  • GCWM may reduce reliance on replay data and diagnose harmful update integration, but easier adaptation could also produce unsafe or misleading behaviors without safety evaluation.Source: A Discussion and Broader impacts · p. 16
  • GCWM operates in parameter-update space for data-free continual integration, unlike distribution-level [68] or preference-level [69] fusion methods targeting cross-model knowledge transfer.Source: A Discussion and Broader impacts · p. 16
  • The work introduces no new safety guarantees, so deployment should combine compatibility-controlled merging with data governance, red-teaming, and downstream safety checks.Source: A Discussion and Broader impacts · p. 16

B Implementation Details of Algorithm

GCWM implements continual merging by selecting active updates, estimating layer-wise geometry conflict, and applying geometry-aware incremental changes relative to the previous merged state. The implementation uses a regularized full-space whitening-and-recoloring transform corresponding to the projected formulation while preserving its geometry.

Implementation overview

Continual update and merge process

  • At each continual step, GCWM selects active updates using either a history-aware memory policy or an anchor-based policy against the previously merged state.For each target layer, it computes shared task geometry, estimates conflict, constructs a shared Wasserstein metric, merges geometrically, and applies only the incremental change.Source: B Implementation Details of Algorithm · p. 16

Transform implementation

  • The implementation uses a regularized full-space whitening-and-recoloring transform that preserves the projected geometry while regularizing its orthogonal complement.The main text presents the corresponding projected formulation for clarity.Source: B Implementation Details of Algorithm · p. 16

C Proof of Theorem 1 · D Proof of Proposition 1

Theorem 1 proves that GCWM’s additional loss relative to plain merging is controlled by shared geometry conflict and gated displacement under stated smoothness and geometry assumptions. Proposition 1 characterizes the gate as threshold-controlled: higher conflict increases the geometry-aware branch’s weight.

C Proof of Theorem 1

  • The theorem is established for the projected shared-metric branch, while the dense-local implementation is treated as an efficiency-oriented special case.The proof compares GCWM with the plain merge at the same continual step to isolate the geometry-aware correction’s effect.Source: C Proof of Theorem 1 · p. 16
  • The proof controls the shared-center mismatch using pairwise geometry conflict measured through Bures distances.The shared Wasserstein barycenter serves as the center, and the weighted mismatch measures how far active task geometries lie from it.Source: C Proof of Theorem 1 · pp. 18–19
  • Its assumptions require local smoothness, adequate projected geometry for interference-relevant directions, and a layer-separable metric-curvature bound.The curvature assumption absorbs cross-layer Hessian couplings into layer-wise constants measured in the implementation-aligned full-space metric.Source: C Proof of Theorem 1 · p. 18
  • Theorem 1 shows that GCWM’s additional loss on previously acquired tasks, relative to plain merging, is bounded by shared geometry conflict and gated merge displacement.The result applies for at least two active updates per analyzed layer under Assumptions 1–3 and the pairwise weights specified in Lemma 1.Source: C Proof of Theorem 1 · p. 20
  • The GCWM gate directly scales the second-order displacement term, making geometry conflict an actionable control signal for update integration.This connects the theorem’s loss bound to the geometry-aware correction applied during merging.Source: C Proof of Theorem 1 · p. 21

D Proof of Proposition 1

  • Proposition 1 proves that conflict below threshold τ yields a weaker geometry-aware correction, whereas conflict above τ assigns the geometry-aware branch a larger weight.This follows from the monotonic sigmoid gate with sharpness parameter κ > 0.Source: D Proof of Proposition 1 · pp. 21–22
  • The gate is implemented through a sigmoid threshold rule controlled by conflict threshold τ and sharpness κ.The proposition’s algebra separates the plain and geometry-aware components before establishing the two conflict regimes.Source: D Proof of Proposition 1 · pp. 21–22

E Analysis Metrics … F.2 Diagnostic Dashboards for Sec. 3

The analysis defines complementary drift, subspace, geometry, and gradient signals, then validates that state-relative geometry conflict robustly tracks forgetting across scales and methods. Diagnostic dashboards further show how these signals relate to retention loss, pairwise compatibility, and predictor complementarity.

E Analysis Metrics

  • The analysis measures parameter drift with update norm, subspace overlap with SAR, geometry conflict with normalized Bures–Wasserstein discrepancy, and gradient alignment with cosine statistics.Gradient diagnostics include mean cosine, the fraction of negative-cosine pairs, and state-relative variants that replace one task update with the current training state.Source: E Analysis Metrics · p. 22

F Additional Empirical Analysis for Sec. 3

  • The additional empirical analysis mirrors four Sec. 3 findings: diagnostic dashboards, step-level and state-relative analysis, pairwise compatibility, and module-level geometry–gradient complementarity.These analyses extend the main empirical evidence rather than introducing a separate evaluation framework.Source: F Additional Empirical Analysis for… · p. 23

F.1 Statistical Confidence for Sec. 3

  • Statistical uncertainty is quantified with 2,000 run-cluster bootstrap resamples and two-sided permutation tests using 3,000 shuffles.Run clusters preserve within-run temporal dependence across continual steps.Source: §F.1 · p. 23
  • Global state-relative geometry gap is the strongest forgetting-associated signal (ρs = −0.59, 95% CI [−0.74, −0.36]), exceeding update norm (−0.48) and active conflict (0.30).Larger models show more negative and better-separated confidence intervals, whereas 0.6B intervals are substantially wider.Source: §F.1 · pp. 23–24
  • Pairwise SAR and geometry conflict are non-redundant (ρs = 0.27), but GC–drop remains near zero (ρs = 0.02, p = 0.485).Geometry signals are strongest for forgetting, whereas gradient diagnostics are stronger for old-task mean and overall score.Source: §F.1 · p. 24, §F.2 · p. 26

F.2 Diagnostic Dashboards for Sec. 3

  • State-relative geometry gaps most closely track forgetting, reaching correlations of −0.67, −0.69, and −0.86 for Qwen3-4B, 8B, and 14B, while GC-drop remains near zero.Under Seq. SFT, state-relative and global geometry gaps evolve with the accumulated model state and more closely track retention loss than abrupt active-pair conflict.Source: §F.2 · p. 26
  • For 1.7B/4B/8B/14B, retention-loss correlations with state-relative conflict are 0.60/0.70/0.75/0.79, while global-gap associations are 0.81/0.68/0.72/0.65.The 0.6B case is noisier, consistent with weaker and less stable small-model correlations.Source: §F.2 · p. 26

F.3 Additional Step-Level and State-Relative Analysis … G.4 Comparison with Non-Continual Model Merging

Across additional analyses, state-relative geometry better predicts forgetting than raw update magnitude, while pairwise conflict alone is insufficient; module-level results distinguish geometry conflict from gradient conflict. In supplementary experiments, GCWM outperforms non-continual merging beyond the smallest scale while using data-free integration and fixed benchmark evaluation procedures.

F.3 Additional Step-Level and State-Relative Analysis

  • State-relative geometry gaps correlate more strongly with forgetting than incremental update norm, reaching ρs = −0.59 globally and −0.67, −0.69, and −0.86 for Qwen3-4B, 8B, and 14B.Active-pair geometry conflict is more method-dependent and remains a weaker step-level predictor.Source: §F.3 · p. 26, Figure 11 · p. 26, (b) EWC · p. 29

(e) EWC

  • Pairwise geometry conflict has only weak association with immediate old-task change, with GC-drop at ρs = 0.01–0.02 and the largest immediate-change predictor correlation only 0.04.Pairwise geometry is more useful for forgetting from the best previous score but remains modest, with mean and maximum GC correlations of 0.16 and 0.17.Source: (e) EWC · pp. 29, 32

F.5 Geometry and Gradient Conflict by Module Family

  • Geometry conflict and gradient conflict emphasize different module families: geometry conflict is concentrated in value and MLP pathways, whereas negative-gradient layers concentrate in attention query/key projections.Across families, q_proj and v_proj have the highest mean geometry conflict at 0.56, while top geometry-conflict layers are led by up_proj at 0.28 and top negative-gradient layers by k_proj at 0.47.Source: §F.5 · pp. 33–34
  • The module-level separation remains method-dependent: Seq. SFT and AIMMerging concentrate geometry conflict in MLP projections, EWC shifts it toward down_proj, and FOREVER emphasizes v_proj.Negative-gradient layers remain consistently concentrated in query/key projections across methods.Source: §F.5 · pp. 33–34

G.1 Experimental Setup Details

  • All data-free integration methods use the same trained task-expert updates without replay data at merge time, while Seq. SFT, EWC, and FOREVER provide sequential, regularized, and replay-based reference pipelines.GCWM’s merge, geometry construction, and scaling analyses run data-free, and merging adds no inference-time cost after integration.Source: §G.1 · pp. 34–35

G.2 Evaluation Prompt

  • Evaluation uses prompt templates for the reported benchmarks, with the full suite covering domain and capability retention, reasoning, mathematics, and code-generation tasks.The evaluation prompt templates are provided in Table 19.Source: §G.1 · p. 34, Table 19 · p. 35

G.3 Performance Context and Data Completeness

  • AIMMerging improves final overall performance at every scale, from 0.271 on Qwen3-0.6B to 0.678 on Qwen3-14B, while final forgetting remains between −0.011 and −0.006.Capability-continual scores are averaged over five independent evaluation runs with fixed benchmark scripts, decoding settings, and task order.Source: §G.1 · p. 34, §G.3 · p. 35, Figure 19 · p. 37

G.4 Comparison with Non-Continual Model Merging

  • +0.21, +3.24, +5.71, +2.64, and +4.28 points are GCWM’s average gains over the strongest non-continual merge across Qwen3 scales, matching or exceeding it on 2/6 to 6/6 benchmarks.DARE’s average score collapses to 15.1% on Qwen3-0.6B, 32.2% on Qwen3-8B, and 29.6% on Qwen3-14B.Source: Table 23 · p. 37, §G.4 · p. 37, Table 24 · p. 38

G.5 Full Domain-Continual Results · G.6 Full Capability-Continual Results

Across all Qwen3 scales, GCWM delivers the strongest non-MTL domain-continual accuracy and the strongest average among data-free capability-continual methods. Its gains are especially consistent at larger scales, while smaller-scale capability interactions remain noisier.

G.5 Full Domain-Continual Results

  • GCWM achieves the best non-MTL MMLU-Pro accuracy across all five Qwen3 scales, improving over the strongest data-free baseline by 0.30, 1.61, 1.19, 0.74, and 1.23 points from 0.6B to 14B.Average accuracy rises from 51.1% for AIMMerging and 51.0% for OPCM to 52.3% for GCWM.Source: §G.5 · p. 38, Table 25 · p. 39, Table 26 · p. 39
  • GCWM also improves full domain-continual performance at small and mid-sized scales, reaching 27.14% on Qwen3-0.6B and 59.43% on Qwen3-4B.On Qwen3-4B, it matches or exceeds the best non-MTL result on 12 of 14 domains.Source: §G.5 · p. 38, Table 27 · p. 39

G.6 Full Capability-Continual Results

  • GCWM is best on capability-continual average at 1.7B, 8B, and 14B, while remaining close to the best baseline at 0.6B and 4B.The full scale sweep is reported in Table 28, with gains summarized in Table 29.Source: §G.6 · p. 40
  • GCWM has the strongest five-scale average among data-free capability-integration methods at 63.94, ahead of OPCM at 61.99, AIMMerging at 60.28, and L&S at 59.53.Its gains over the strongest data-free baseline are -0.11, +5.78, -0.18, +1.61, and +1.39 points from 0.6B to 14B.Source: §G.6 · p. 40
  • At 14B, GCWM leads data-free baselines on GPQA-Diamond, GSM8K, HumanEval, and MMLU-Pro, while remaining competitive on MATH-500 and MBPP.These larger-scale gains indicate clearer capability transfer benefits under geometry-compatible integration.Source: §G.6 · p. 40
  • At smaller scales, capability interactions are noisier: GCWM is strongest on HumanEval and MBPP at 0.6B but trails AIMMerging slightly in average score.This pattern is consistent with state-relative geometry findings that compatibility-aware integration yields more stable gains as model capacity increases.Source: §G.6 · p. 40

H Additional Ablation Study

Capability-continual ablations on Qwen3-1.7B and Qwen3-8B show that both GCWM components improve aggregate performance. The gate and Wasserstein shared metric are especially important when heterogeneous math, code, and knowledge updates interact.

Ablation design

  • The ablations isolate GCWM’s merge-time components by removing conflict-conditioned gating or replacing the Wasserstein barycenter while holding experts and evaluation fixed.Without the gate, the geometry-aware branch is applied uniformly; without WB, a simpler shared metric is used.Source: H Additional Ablation Study · p. 40

Interpretation

  • The gate controls how strongly geometry-aware integration is trusted, while the Wasserstein shared metric provides a common space for combining heterogeneous updates.Together, the results support compatibility-controlled merging rather than uniform application of one integration rule.Source: H Additional Ablation Study · pp. 40, 42

Aggregate ablations

  • Full GCWM reaches 58.26% versus 56.68% without the gate and 56.95% without the Wasserstein barycenter on Qwen3-1.7B; on Qwen3-8B, it reaches 72.54% versus 67.93% and 68.81%.The larger model shows stronger component dependence in capability-continual merging.Source: H Additional Ablation Study · p. 40, Table 30 · p. 42

Benchmark breakdown

  • On Qwen3-8B, the gate adds 14.0 points on MATH-500, while the Wasserstein barycenter supports GSM8K, HumanEval, MMLU-Pro, and GPQA-Diamond.Effects vary by benchmark: some ablations improve individual tasks, but both substantially reduce MMLU-Pro and GPQA-Diamond on Qwen3-1.7B.Source: H Additional Ablation Study · p. 40, Table 31 · p. 42

I Runtime and Memory Profiling … 2. Limitations

This section covers GCWM’s runtime and memory costs, hyperparameter sensitivity, and the study’s methodological, computational, reproducibility, and societal-impact limitations. It also documents the absence of crowdsourcing or human-subject research and clarifies the limited role of LLMs in writing, editing, and formatting.

I Runtime and Memory Profiling

  • 40.5 ± 19.7 minutes and 7.8 ± 3.4 GB characterize Qwen3-8B merging, versus 76.2 ± 34.8 minutes and 11.7 ± 4.7 GB for matched Qwen3-14B steps.Cost increases with active updates, while SPD operations remain in projected rank-r or union-rank spaces; SVD/metric preparation and inner optimization dominate runtime.Source: I Runtime and Memory Profiling · p. 42, Table 32 · p. 43

J Hyperparameter Sensitivity

  • GCWM is stable under moderate geometry and gate settings, with average-score changes of 1.6, 0.9, and 1.4 points across τ, r, and κ sweeps.Energy 0.95 reaches 72.2%, whereas ηt = 1.0 collapses performance to 34.3%, motivating conservative coefficient schedules.Source: J Hyperparameter Sensitivity · p. 43, Table 34 · p. 44

NeurIPS Paper Checklist

  • The checklist states that the paper’s claims match its theoretical and experimental support, while acknowledging benchmark scope, geometry-conflict interpretation, replay tradeoffs, and offline compute overhead.The limitations discussion also covers assumptions, robustness, computational efficiency, and the restricted scope of empirical evaluation.Source: NeurIPS Paper Checklist · p. 45, §2 · p. 45

2. Limitations

  • The paper reports reproducibility through documented setups, metrics, baselines, evaluation protocols, implementation details, and anonymized code and data availability materials.It also reports statistical confidence for key claims using run-cluster bootstrap intervals and permutation tests.Source: §2 · pp. 46–47
  • The broader-impact discussion notes possible misuse of easier post-training and calls for downstream safety checks, while no new high-risk foundation model or scraped dataset is released.Existing assets are credited and used under standard research terms, with explicit license information planned for the Appendix.Source: §2 · pp. 49–50
  • The work involves no crowdsourcing tasks or human-subject experiments, making participant instructions, screenshots, and compensation details not applicable.The paper also states that human-subject research requiring IRB review was not conducted.Source: §2 · pp. 50–51
  • No IRB approval or equivalent review was required because the study does not involve human-subject research.The supplied justification explicitly marks this requirement as [N/A].Source: §2 · p. 51
  • LLMs were used only for writing, editing, and formatting, so their usage was not an important, original, or non-standard component of the core method.Accordingly, the paper marks the LLM-usage declaration as [N/A].Source: §2 · p. 51

Related papers

Loading 2605.09608v1…