PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Yuanyi Wang, Yifan Yang, Su Lu, Yanggan Gu, Pengkai Wang, Wenjun Wang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Jialun Cao, Shing-Chi Cheung, Hongxia Yang

arXiv:2605.09608v1cs.LGcs.IT

TL;DR

LLM continual post-training lacks clear criteria for when sequential updates transfer capabilities rather than cause catastrophic forgetting. This paper analyzes task-update geometry and introduces GCWM, finding that state-relative geometry conflict explains forgetting and improves data-free update integration across continual settings.

  • Problem

    Continual post-training lacks clear criteria for when sequential updates transfer capabilities rather than cause catastrophic forgetting from interference between updates.

  • Method

    The paper models task updates through covariance geometry and uses state-relative Bures–Wasserstein conflict to gate geometry-aware correction in data-free GCWM merging.

  • Results

    GCWM consistently improves retention and final performance over data-free baselines across settings and Qwen3 scales, reaching 62.6 versus 56.8 average performance at 1.7B.

  • Takeaways & Limitations

    State-relative geometry conflict is both an explanatory signal for forgetting and a practical control signal for sequential update integration.

  • Takeaways & Limitations

    The evidence focuses on Qwen3-scale open LLMs and selected continual tasks, and geometry conflict is not established as causally necessary for all forgetting.

Abstract

from arXiv · show

Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.

1 Introduction

The introduction frames catastrophic forgetting as interference among sequential parameter updates and argues that task geometry can explain and control when updates transfer or conflict. It presents GCWM, a data-free, compatibility-controlled merging method that improves retention and final performance across Qwen3 scales without replay data.

  • Motivation: Continual post-training extends LLMs through sequential stages targeting new domains, skills, or behaviors, but new learning can undermine previously acquired knowledge.This failure is identified as catastrophic forgetting and is often driven by interference between sequential parameter updates.
  • Motivation: Existing solutions use sequential fine-tuning, replay, regularization, or model merging, yet lack a principled account of task compatibility.Consequently, they provide limited guidance about when integrating a new update is beneficial or harmful.
  • Task Geometry: The paper represents each post-training task by its parameter update and studies induced covariance geometry, capturing update magnitude, subspaces, and spectral structure.This task-geometry view addresses what drives forgetting, when capabilities transfer or interfere, and how compatibility can control update integration.
  • Core Finding: Forgetting is a state-relative update-integration failure caused by covariance geometries misaligning with the evolving model state, whereas transfer occurs when updates remain compatible.The analysis compares geometry conflict with update norm, subspace alignment ratio, and gradient conflict across Qwen3 scales and continual strategies.
  • Method and Results: GCWM builds a shared Wasserstein metric via Gaussian Wasserstein barycenters and gates geometry-aware correction using geometry conflict for data-free update integration.Across Qwen3 models from 0.6B to 14B, it consistently improves retention and final performance over data-free baselines without replay data.

2 Preliminary

The paper formulates LLM continual post-training as sequential adaptation through task-specific parameter updates and analyzes forgetting through task-induced covariance geometry. It defines geometry conflict using normalized Bures–Wasserstein distance and positions it as a control-oriented compatibility signal beyond standard diagnostics.

  • Problem Setup: Continual post-training adapts a pretrained LLM through sequential tasks introducing new domains, skills, or behaviors, represented by parameter-efficient or full-model task updates.These task updates are the basic objects for analyzing continual post-training.
  • Task Geometry: Task geometry captures dominant directions of layer-wise update matrices, because update norm alone cannot distinguish updates affecting different subspaces.Two updates with similar magnitude can therefore induce different forgetting behavior.
  • Geometry Conflict: Geometry conflict measures discrepancy between task-induced geometries in a shared basis using normalized Bures–Wasserstein distance.Lower geometry-conflict values indicate more compatible task-induced geometries.
  • Compatibility Diagnostics: The study compares geometry conflict with update norm, subspace alignment ratio (SAR), and gradient cosine conflict, while state-relative variants replace one task update with the current continual-training state.Full metric definitions and aggregation details are provided in Appendix E.
  • Related Work: Prior continual post-training approaches include sequential fine-tuning, replay-based methods, continual model merging, and compatibility signals based on parameter discrepancy, gradient alignment, or subspace overlap.The paper distinguishes its geometry conflict as a method-native control signal rather than a largely diagnostic compatibility measure.

3 What Governs Forgetting in Continual Post-Training?

Forgetting is best explained as a state-relative geometry mismatch rather than parameter drift alone. Geometry conflict complements SAR and gradient conflict, revealing distinct compatibility and failure modes across continual post-training updates.

  • 3.1 What Governs Forgetting?: State-relative geometry mismatch best tracks continual forgetting, while update norm provides only a coarse drift baseline.The analysis compares update norm, SAR, gradient conflict, and geometry conflict across Qwen3 models and four continual-training strategies.
  • 3.1 Pairwise Compatibility: SAR and geometry conflict are related but non-redundant, with global rank association ρs = 0.27; similar SAR can correspond to very different geometry conflict.SAR captures where updates overlap, whereas geometry conflict captures whether their induced covariance geometries are compatible.
  • 3.2 State-Relative Geometry: State and global geometry gaps follow retention-loss growth more closely than active-pair conflict, especially for Qwen3 models from 1.7B to 14B.Under Seq. SFT, active-pair conflict fluctuates across steps, whereas state and global gaps track the evolving model state.
  • 3.2 State-Relative Geometry: 0.68/0.70 are the Seq. SFT state/global correlations, compared with 0.40/0.42 for EWC; replay and merging weaken these signals by compressing forgetting variance.The method-level heatmap reports state/global signals of 0.68/0.70 for Seq. SFT and 0.40/0.42 for EWC.
  • 3.3 Complementary Failure Modes: Geometry conflict does not duplicate gradient conflict: top conflict layers concentrate in up_proj, gate_proj, v_proj, and down_proj.The analysis defines top-layer share, minimum gradient cosine, and negative-cosine ratio to compare module-level geometry and gradient signals.

4 Geometry Conflict Wasserstein Merging

GCWM is a data-free, compatibility-controlled merging method that estimates layer-wise geometry conflict, constructs a shared Wasserstein metric, and gates geometry-aware correction. It applies only incremental changes and provides bounds linking its relative loss effect and update displacement to geometry conflict.

  • Incremental integration: At each continual step, GCWM computes a merged proposal from active updates and applies only its change relative to the previous merged state.The active set may include the current update, historical updates, or the previous merged state.
  • Geometry conflict estimation: GCWM represents task updates through layer-wise covariance geometry and uses normalized Bures–Wasserstein discrepancies to estimate pairwise geometry conflict.The conflict score compares projected task-induced geometries and indicates their compatibility.
  • Geometry conflict estimation: A layer-wise gate converts aggregated geometry conflict into actionable control of geometry-aware correction, with threshold τ and sharpness κ determining gate behavior.Low-conflict layers receive weaker correction, whereas high-conflict layers receive stronger correction.
  • Wasserstein merging: GCWM constructs a shared metric with Gaussian Wasserstein barycenters, whitens projected updates, applies weighted WUDI, and recolors the merged result before blending it with a plain merge.The barycenter defines the local metric in which active updates are aligned before merging.
  • Conflict-controlled analysis: Theorem 1 bounds the additional loss from GCWM relative to plain merging by shared geometry conflict and metric displacement.The analysis assumes local smoothness, projected-geometry adequacy, and layer-wise metric-curvature conditions.

5 Experiments

Experiments evaluate GCWM as a data-free update-integration method under domain and capability shifts, using Qwen3 models from 0.6B to 14B. GCWM achieves the strongest non-MTL domain performance and the best data-free capability-continual averages reported.

  • Experimental setup: GCWM is evaluated under domain-continual and capability-continual shifts, with comparisons centered on data-free merging baselines and sequential, regularized, and replay-based reference pipelines.The models are Qwen3 backbones at 0.6B, 1.7B, 4B, 8B, and 14B; the domain sequence contains 14 tasks, while capability training uses math and code updates.
  • Domain-continual post-training: GCWM gives the strongest non-MTL overall performance on the 14-domain MMLU-Pro sequence at 1.7B, 8B, and 14B.MTL serves as a joint-training upper-bound reference, while the main comparison is among data-free update-integration methods.
  • Domain-continual post-training: GCWM improves over AIMMerging on 12/14 domains at 1.7B, 10/14 domains at 8B, and 9/14 domains at 14B.These results support geometry conflict as a practical control signal for data-free continual update integration.
  • Capability-continual post-training: 62.6 is GCWM’s best data-free average at 1.7B, versus 56.8 for OPCM, a +5.78-point improvement, with GCWM leading on all six capability benchmarks.The benchmarks cover knowledge, math, and code, including GPQA-Diamond, MMLU-Pro, GSM8K, MATH-500, HumanEval, and MBPP.
  • Capability-continual post-training: 74.3 is GCWM’s strongest data-free average at 14B in the capability-continual evaluation.Table 2 evaluates Qwen3-1.7B and 14B across knowledge, math, and code benchmarks.

6 Conclusion

The paper frames forgetting in LLM continual post-training as a state-relative update-integration failure identified through task geometry. Harmful updates arise when task-induced covariance geometries become incompatible with the evolving model state.

  • Conclusion: The study analyzes continual post-training through task geometry across model scales and continual strategies.Its analysis considers update norm, subspace alignment, gradient conflict, and geometry conflict.
  • Conclusion: Forgetting is a state-relative update-integration failure caused by incompatibility between task-induced covariance geometries and the evolving model state.The conclusion identifies this incompatibility as the condition under which harmful update steps occur.
  • Conclusion: Raw drift and isolated pairwise compatibility are insufficient to explain forgetting.The conclusion attributes this limitation to the state-relative nature of update integration.

Limitations · A Discussion and Broader impacts

The study’s evidence is limited to Qwen3-scale open LLMs and selected domain- and capability-continual tasks, so geometry conflict is presented as an explanatory and control signal rather than a universally causal account of forgetting. GCWM may reduce reliance on replay data, but deployment still requires safety evaluation, data governance, red-teaming, and downstream checks.

  • Limitations: The experiments focus on Qwen3-scale open LLMs and continual tasks built from public reasoning, knowledge, math, and code benchmarks.The state-relative geometry signal is consistent across scales and methods within this scope.
  • Limitations: State-relative geometry should be treated as an explanatory and control signal, not proof of causal necessity for every form of forgetting.This qualification limits how broadly the paper’s interpretation should be generalized.
  • Limitations: GCWM is data-free at merge time and is most relevant when historical data are unavailable or inaccessible.Its practical value is tied to continual update integration without replay data.
  • A Discussion and Broader impacts: GCWM may make continual LLM adaptation more practical by reducing reliance on replay data and diagnosing harmful update integration.These benefits arise from compatibility-controlled, data-free merging.
  • A Discussion and Broader impacts: Easier post-training can also adapt models toward unsafe or misleading behaviors without appropriate safety evaluation.The broader impact therefore includes potential misuse of more accessible adaptation procedures.
  • A Discussion and Broader impacts: GCWM operates in parameter-update space for data-free continual update integration, unlike distribution- or preference-level fusion methods targeting cross-model knowledge transfer.The distinction concerns both the operating space and the primary objective of the compared fusion approaches.
  • A Discussion and Broader impacts: The work introduces no new safety guarantees for compatibility-controlled merging.Safety claims should not be inferred from the proposed forgetting-control mechanism.
  • A Discussion and Broader impacts: Before deployment, model developers should combine compatibility-controlled merging with standard data governance, red-teaming, and downstream safety checks.These safeguards are recommended because the method does not itself establish deployment safety.

B Implementation Details of Algorithm

GCWM builds an active set of task updates, estimates their geometry conflict, and performs incremental geometry-aware merging relative to the previous merged state. Its implementation uses a regularized full-space whitening and recoloring transform corresponding to the projected formulation.

  • Active-set construction: At each continual step, GCWM selects updates through either a history-aware memory policy or an anchor-based policy.The history-aware policy retains previous task updates, whereas the anchor-based policy merges the current task against the previously merged state.
  • Geometry-aware merging: For each target layer, GCWM computes shared task geometry, estimates geometry conflict, constructs a shared Wasserstein metric, and performs geometry-aware merging.The method applies only the incremental change relative to the previous merged state.
  • Whitening and recoloring: The implementation uses a regularized full-space whitening and recoloring transform that preserves the projected geometry while regularizing the orthogonal complement.The main text presents a projected formulation for clarity.
  • Algorithm overview: The full continual update and merge process is summarized in Algorithm 1.All task updates are constructed relative to the same pretrained model.

C Proof of Theorem 1 · D Proof of Proposition 1

Theorem 1 shows that GCWM’s additional loss over plain merging is controlled by shared geometry conflict and gated displacement under stated smoothness and geometry assumptions. Proposition 1 establishes that the gate weakens correction below the conflict threshold and increases geometry-aware weighting above it.

  • C Proof of Theorem 1: The proof compares GCWM with the plain merge at the same continual step, isolating the effect of geometry-aware correction in the projected shared-metric branch.The dense-local implementation is treated as an efficiency-oriented special case.
  • C Proof of Theorem 1: GCWM defines its correction relative to the plain merge and measures displacement using an implementation-aligned full-space regularized metric.The metric corresponds to the full-space whitening and recoloring operators used in implementation.
  • C Proof of Theorem 1: The shared Wasserstein barycenter provides a center whose mismatch measures how far active task geometries lie from their shared center.Lemma 1 controls this center mismatch using pairwise geometry conflict.
  • C Proof of Theorem 1: Under Assumptions 1–3, GCWM’s additional loss on a previously acquired task relative to plain merging is bounded by shared geometry conflict and gated merge displacement.The theorem is stated for at least two active updates per analyzed layer and uses pairwise weights from Lemma 1.
  • C Proof of Theorem 1: The GCWM gate directly scales the second-order displacement term, making geometry conflict an actionable control signal.This connects the theorem’s loss control to the algorithm’s gated integration mechanism.
  • D Proof of Proposition 1: The gate uses a sigmoid with sharpness parameter κ and conflict threshold τ to determine the relative weighting of plain and geometry-aware updates.The threshold characterization follows from sigmoid monotonicity and σ(0) = 1/2.
  • D Proof of Proposition 1: When layer conflict g^(ℓ)_t is below τ, GCWM applies a weaker geometry-aware correction.This is the low-conflict regime established by the proposition’s threshold characterization.
  • D Proof of Proposition 1: When layer conflict g^(ℓ)_t exceeds τ, the geometry-aware branch receives a larger weight.This establishes the high-conflict regime claimed in Proposition 1.

E Analysis Metrics … F.2 Diagnostic Dashboards for Sec. 3

The paper uses four signal families—parameter drift, subspace overlap, geometry conflict, and gradient compatibility—to diagnose continual post-training. Additional analyses validate these signals statistically and show that state-relative geometry more closely tracks forgetting than coarse drift or pairwise conflict alone.

  • E Analysis Metrics: The analysis measures parameter drift with update norm, subspace overlap with SAR, geometry conflict with normalized Bures–Wasserstein discrepancy, and gradient compatibility with cosine similarity.State-relative variants replace one task update with the current continual-training state.
  • F Additional Empirical Analysis for Sec. 3: The empirical analysis covers diagnostic dashboards, step-level and state-relative analysis, pairwise compatibility, and module-level geometry–gradient complementarity.These four analyses mirror the empirical findings in Sec. 3.
  • F.1 Statistical Confidence for Sec. 3: Statistical uncertainty is quantified with 2,000 run-cluster bootstrap resamples and two-sided permutation p-values from 3,000 shuffles.Run clusters preserve within-run temporal dependence across continual steps.
  • F.1 Statistical Confidence for Sec. 3: At the global step level, global state-relative geometry gap is the strongest forgetting-associated signal (ρs = −0.59, 95% CI [−0.74, −0.36]).It is stronger than update norm (−0.48, CI [−0.61, −0.20]) and active conflict (0.30, CI [0.00, 0.58]).
  • F.1 Statistical Confidence for Sec. 3: Larger models show more negative and better-separated state/global confidence intervals, while 0.6B intervals are substantially wider.At the pairwise level, SAR–GC is non-redundant (ρs = 0.27), whereas GC–drop remains near zero (ρs = 0.02, p = 0.485).
  • F.2 Diagnostic Dashboards for Sec. 3: Diagnostic dashboards show global state-active gap has the strongest step-level association with forgetting (ρs = −0.59), exceeding update norm (ρs = −0.48) and active-pair geometry conflict (ρs = 0.30).SAR and geometry conflict are related but non-redundant, while pairwise geometry conflict alone remains weak for immediate selective forgetting.
  • F.2 Diagnostic Dashboards for Sec. 3: Under Seq. SFT, state-relative and global geometry gaps evolve with the accumulated model state and more closely track retention loss than abrupt local active-pair conflict changes.For 1.7B/4B/8B/14B, retention-loss correlations with state-relative conflict are 0.60/0.70/0.75/0.79, versus global-gap associations of 0.81/0.68/0.72/0.65; 0.6B is noisier.

F.3 Additional Step-Level and State-Relative Analysis … G.4 Comparison with Non-Continual Model Merging

The analyses show that state-relative geometry better explains forgetting than raw update magnitude, while pairwise conflict alone is insufficient; module-level results distinguish geometry mismatch from gradient opposition. Additional experiments establish evaluation completeness and show that GCWM is more robust than non-continual merging, especially beyond the smallest model.

  • F.3 Additional Step-Level and State-Relative Analysis: State-active geometry gap correlates more strongly with forgetting than incremental update norm, reaching −0.59 globally and −0.67, −0.69, and −0.86 at larger scales.Raw update magnitude captures coarse drift but does not consistently explain retention across methods.
  • (a) Seq. SFT; (b) EWC: Seq. SFT shows the clearest state-relative pattern, with state and global gaps correlating with forgetting at ρs = −0.68 and −0.70, versus −0.23 for update norm.For EWC, active-pair geometry conflict and gradient conflict are more pronounced, with Active GC at ρs = 0.51 and minimum gradient cosine at ρs = 0.54.
  • F.3 Additional Step-Level and State-Relative Analysis; F.5 Geometry and Gradient Conflict by Module Family: Across targets, the global gap is the strongest geometry-based forgetting signal at ρs = −0.59, while minimum gradient cosine reaches ρs = 0.46 with old-task mean and ρs = 0.33 with overall score.These results support geometry conflict as a compatibility signal rather than treating update magnitude as sufficient.
  • F.4 Additional Pairwise Compatibility Analysis; (e) EWC: Across 1,820 task pairs, SAR and geometry conflict have moderate global association at ρs = 0.27, but geometry conflict is only weakly correlated with immediate old-task change at ρs = 0.02.Pairwise conflict identifies compatibility regimes, but forgetting is not determined by isolated task pairs alone.
  • F.4 Additional Pairwise Compatibility Analysis; (e) EWC: Pairwise predictors have limited explanatory power for immediate old-task change, with a largest absolute Spearman correlation of only 0.04; mean and max geometry conflict correlate with forgetting at ρs = 0.16 and 0.17.The largest Qwen3-4B EWC drops cluster around Math, including Math→History (−0.129), Math→Economics (−0.123), and Math→Chemistry (−0.114).
  • F.5 Geometry and Gradient Conflict by Module Family: Geometry conflict and gradient conflict emphasize different module families: geometry conflict is broad across attention and MLP projections, while negative-gradient layers concentrate in query/key modules.Mean geometry-conflict values are 0.56 for both q_proj and v_proj, while top negative-gradient layers account for 0.47 in k_proj and 0.38 in q_proj.
  • G Additional Experiments for Sec. 5; G.1 Experimental Setup Details; G.2 Evaluation Prompt; G.3 Performance Context and Data Completeness: All final-analysis benchmark runs contain 14 continual steps with no missing values, and capability-continual performance scores are averaged over five independent evaluation runs.Data-free integration methods use the same trained-expert inputs, while MTL is treated as a joint-training upper-bound reference.
  • G.4 Comparison with Non-Continual Model Merging: DARE’s average score collapses to 15.1% on Qwen3-0.6B, 32.2% on Qwen3-8B, and 29.6% on Qwen3-14B, while GCWM is more effective beyond the smallest scale.DARE also reaches 0.8% on MBPP at 8B and 0.4% on MATH-500 at 14B, indicating brittleness without continual compatibility modeling.

G.5 Full Domain-Continual Results · G.6 Full Capability-Continual Results

Across all five Qwen3 scales, GCWM delivers the strongest non-MTL domain-continual accuracy and the strongest average among data-free capability-integration methods. Its gains are largest and most consistent at several larger scales, while smaller-scale capability interactions remain noisier.

  • G.5 Full Domain-Continual Results: The full domain-continual results span all five Qwen3 scales, with complete overall, gain, and additional per-domain results reported in Tables 25–27 alongside Table 1.MTL is used as an upper-bound reference in the reported tables.
  • G.5 Full Domain-Continual Results: GCWM improves over the strongest data-free domain baseline by +0.30, +1.61, +1.19, +0.74, and +1.23 points across Qwen3-0.6B, 1.7B, 4B, 8B, and 14B.Average overall accuracy rises from 51.1% for AIMMerging and 51.0% for OPCM to 52.3% for GCWM.
  • G.5 Full Domain-Continual Results: GCWM achieves the best non-MTL overall score on Qwen3-0.6B at 27.14% and improves over all data-free baselines on Qwen3-4B at 59.43%.On Qwen3-4B, it matches or exceeds the best non-MTL result on 12 of 14 domains.
  • G.6 Full Capability-Continual Results: GCWM has the strongest five-scale average among data-free capability-integration methods at 63.94, ahead of OPCM at 61.99, AIMMerging at 60.28, and L&S at 59.53.It is best on overall average at 1.7B, 8B, and 14B, while remaining close to the best baseline at 0.6B and 4B.
  • G.6 Full Capability-Continual Results: The capability-continual gain over the strongest data-free baseline is -0.11, +5.78, -0.18, +1.61, and +1.39 points on 0.6B, 1.7B, 4B, 8B, and 14B, respectively.Full results and scale-wise gains are reported in Tables 28–29 together with Table 2.
  • G.6 Full Capability-Continual Results: On Qwen3-14B, GCWM leads data-free baselines on GPQA-Diamond, GSM8K, HumanEval, and MMLU-Pro while remaining competitive on MATH-500 and MBPP.At 0.6B, it is strongest on HumanEval and MBPP but slightly trails AIMMerging in average score.
  • G.6 Full Capability-Continual Results: Figure 20 compares full GCWM with variants that remove conflict gating or replace the Wasserstein barycenter.The figure provides a capability ablation breakdown.

H Additional Ablation Study

The ablations show that GCWM’s geometry-conflict gate and Wasserstein shared metric both contribute to capability-continual performance. Their benefits are scale- and benchmark-dependent, with stronger aggregate effects on Qwen3-8B.

  • Ablation design: GCWM ablations isolate the geometry-conflict gate and Wasserstein shared metric while holding task experts and evaluation protocol fixed.Removing the gate applies geometry-aware integration uniformly; removing the Wasserstein barycenter replaces it with a simpler shared metric.
  • Aggregate results: 58.26% is full GCWM’s average on Qwen3-1.7B, versus 56.68% without the gate and 56.95% without the Wasserstein barycenter.Both components improve aggregate performance at this scale.
  • Aggregate results: 72.54% is full GCWM’s average on Qwen3-8B, versus 67.93% without the gate and 68.81% without the Wasserstein barycenter.The larger-scale capability setting shows a larger gap between full GCWM and either ablation.
  • Benchmark breakdown: Ablation effects vary across heterogeneous benchmarks: removing either component can improve some metrics but substantially reduces MMLU-Pro and GPQA-Diamond on Qwen3-1.7B.Removing the gate can improve MATH-500, while removing the Wasserstein barycenter can improve HumanEval.
  • Benchmark breakdown: On Qwen3-8B, the gate is especially important for MATH-500, producing a +14.0-point advantage over the no-gate variant.The supplied passage identifies this benchmark-specific difference but truncates the corresponding Wasserstein-barycenter result.
  • Merge mechanism: The gate controls trust in geometry-aware integration, while the Wasserstein shared metric provides the common space for combining heterogeneous task updates.Together, they define GCWM’s merge-time integration mechanism.

I Runtime and Memory Profiling … 2. Limitations

The paper reports GCWM’s merge-time feasibility, hyperparameter robustness, and reproducibility evidence, while documenting limitations involving benchmark scope, interpretation, compute overhead, and societal impacts. GCWM incurs one-time integration cost without inference-time overhead, and its sensitivity is moderate for key geometry and gating parameters.

  • I Runtime and Memory Profiling: GCWM’s overhead is incurred once during data-free model integration, adds no inference-time cost, and remains feasible because expensive matrix operations use low-rank projected SPD matrices.The dominant costs are summarized as O(mdoutdinr) + O(Ir3) + O(mdoutr2), up to implementation constants and merge-optimizer iterations.
  • I Runtime and Memory Profiling: GCWM’s average merge step takes 40.5 ± 19.7 minutes and 7.8 ± 3.4 GB peak GPU memory on Qwen3-8B, versus 76.2 ± 34.8 minutes and 11.7 ± 4.7 GB on Qwen3-14B.The Qwen3-14B figures use two matched profiled steps, while Qwen3-8B uses three.
  • J Hyperparameter Sensitivity: Changing τ over {0.08, 0.12, 0.18}, r over {8, 16, 32, 64}, and κ over {5, 10, 20} changes average score by only 1.6, 0.9, and 1.4 points, respectively.The sweep varies one parameter at a time while keeping the remaining GCWM configuration fixed.
  • J Hyperparameter Sensitivity: Energy 0.95 gives the best average score of 72.2%, while default energy 0.90 is more conservative on MMLU-Pro and GPQA.The reported trade-off concerns the retained geometry-energy threshold.
  • NeurIPS Paper Checklist: The checklist states that the paper’s abstract and introduction accurately reflect its state-relative geometry explanation and GCWM as a data-free integration method.These claims are described as supported by the empirical and theoretical sections.
  • 2. Limitations: Limitations discussed in Appendix 6 include benchmark scope, interpretation of geometry conflict, replay-versus-data-free tradeoffs, and offline compute overhead.The checklist also records theoretical assumptions, statements, and proofs in the main text and appendices.
  • 2. Limitations: The broader-impact discussion covers data-free continual adaptation, possible misuse of easier post-training, and the need for downstream safety checks.The submission studies existing model families and benchmarks rather than releasing a new high-risk foundation model or scraped dataset artifact.
Loading 2605.09608v1…