PaperlayerView exact revision on arXiv

Source-linked AI summary

A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

Lei Yang, Siyu Ding, Deyi Xiong

arXiv:2606.02398v1cs.LGcs.CL

TL;DR

This paper addresses why sequential multi-domain RL can cause selective cross-domain degradation even when full-model gradients are nearly orthogonal. It develops a local route-level perturbation account and finds that short refreshes and sparse proxy rollback can recover damaged performance while largely preserving other domains.

  • Problem

    Sequential multi-domain RL can selectively degrade earlier-domain performance, while catastrophic forgetting and global gradient conflict do not explain interference when full-model gradients are nearly orthogonal.

  • Method

    The paper analyzes sparse RL edits, shared active computation routes, and update-direction alignment, then models interference as second-order damage concentrated in a low-dimensional shared conflict subspace.

  • Results

    66.04 Math performance after a brief Re-Math refresh recovered performance from 57.66 while largely preserving Code, QA, and CW, producing the best overall average score of 66.39.

  • Takeaways & Limitations

    Localized route-level interference offers a mechanistic account of cross-domain damage, while short refresh and sparse proxy rollback support selective recovery with limited collateral damage.

  • Takeaways & Limitations

    The conflict-subspace intervention uses a coarse, basis-aligned proxy that does not directly estimate a latent rotated subspace or cover normalization, residual-stream, and higher-order cross-module interactions.

Abstract

from arXiv · show

Reinforcement learning (RL) post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.

1 Introduction

Sequential multi-domain RL can selectively damage earlier-domain performance despite near-orthogonal global gradients. The paper attributes this interference to sparse edits interacting on shared routes and shows that targeted refresh and rollback support localized recovery.

  • Sequential Code →Math →QA →CW training raises Math to 66.49 after Math training but later reduces it to 57.66, while Code and QA remain relatively stable.
  • The paper shifts analysis from full-model objective conflict to active computation routes, combining edit location, route overlap, and directional agreement.
  • A local perturbation model treats later-domain harm as a second-order damage term concentrated in a low-dimensional shared active conflict subspace.
  • 66.04 Math performance after a brief Re-Math refresh largely preserves Code, QA, and CW, producing the best overall average score of 66.39.
  • The proposed evidence combines short-refresh recovery with training-free targeted rollback on proxy conflict coordinates for the Math–QA pair.

2 Related Work

Prior work frames cross-domain degradation through RL post-training, continual-learning and forgetting accounts, gradient-based multi-task optimization, and localized model changes. This paper instead investigates where interference resides when global gradients appear nearly orthogonal.

  • RL post-training research spans human-preference feedback, summarization, process supervision, mathematical reasoning, reasoning RL, and recent multi-domain methods.
  • The present work asks where cross-domain interference resides, why it is selective, and why a short domain refresh restores performance.
  • Continual-learning research discusses forgetting and alignment or policy shifts, while SFT–RL comparisons examine the degree of model drift and reasoning transfer.
  • Multi-task optimization provides a global perspective through gradient balancing and gradient surgery.
  • Task-vector and mechanistic studies localize transfer, interference, knowledge, skills, and task-relevant features to compact parameter or neuron subsets.

3 Empirical Study Setup

The study uses a controlled four-domain RL setting with sequential and mixed-training comparisons. It evaluates sparse parameter changes and domain performance across independent benchmarks.

  • The experiments use Qwen3-4B-Thinking-2507 across mathematical reasoning, code generation, question answering, and creative writing.
  • Training uses GRPO in VeRL with shared hyperparameters except for domain, initialization checkpoint, and domain-specific reward functions.
  • Single-domain experts and the sequential Code →Math →QA →CW chain are analyzed using cumulative and checkpoint-relative parameter changes.
  • Mixed-training baselines include naive joint training and CGPO, which combines domain-wise batch updates with a specified learning rate.
  • Final evaluation uses independent benchmarks for Math, Code, QA, and CW after selecting checkpoints by current-domain validation performance.

4 Structural Evidence for Localized Cross-Domain Interference

Structural analyses show that global gradients and direct neuron co-editing are insufficient explanations for interference. Sparse edits interact through shared active routes, with update direction determining synergy or conflict.

  • Math–QA global gradient cosine stays near zero, while layer- and module-level analysis reveals localized conflict and synergy.
  • Approximately 77%–89% of parameters in single-domain experts change by less than 10^-7 absolutely and 10^-3 relatively, indicating sparse, mild perturbations.
  • Top-changed neurons overlap weakly across domains, with average Jaccard coefficients below 0.19 for every domain pair.
  • Reasoning domains share more top-active neurons than changed neurons, so sparse updates can remain functionally coupled through shared active routes.
  • Directional alignment on shared neurons differs by pair: Code–Math is positive, whereas Math–QA is negative in L3–6 and positive in L14–21.
  • Together, the findings motivate modeling interference as localized conflict on shared active computation routes rather than as a full-model phenomenon.

5 Theory: Local Recoverability under Sparse Low-Dimensional Interference

The theory explains selective interference as localized second-order damage from later-domain updates in a low-dimensional shared conflict subspace, and predicts recovery through short domain refreshes.

  • Structural motivation: Cross-domain interference is not well explained by full-model gradient conflict because domains can have nearly orthogonal gradients while sharing active routes.Sparse, small-magnitude edits with weak neuron overlap can still interact through reused computation routes.
  • Second-order local damage: A later-domain update harms an earlier domain mainly through second-order curvature after the earlier-domain checkpoint is approximately stationary.The damage is large when the update enters high-curvature directions of the earlier-domain objective.
  • Low-dimensional conflict subspace: The harmful component is modeled as concentrated in a low-dimensional shared active conflict subspace, so low edit overlap does not imply low interference.The leading damage depends on the later update’s projection onto this subspace, with controlled residual terms.
  • Short refresh: A short refresh on the earlier domain contracts the harmful component geometrically under positive restricted curvature and weak coupling from the orthogonal complement.This provides a local recovery mechanism without reversing the entire later-domain update.
  • Selective recovery: Short refresh can restore the damaged domain while keeping collateral damage to other domains bounded under global near-orthogonality.The theory predicts selective recovery rather than full retraining or wholesale rollback.

6 Task-Level Validation and Direct Intervention

Task-level results show selective Math degradation after sequential training and substantial recovery after a brief Re-Math refresh, while a sparse coordinate-proxy rollback independently restores part of the Math loss.

  • Overall validation: Together, refresh and sparse proxy rollback support localized selective interference and recovery without full retraining.The two interventions validate the theory at task and weight-space levels.
  • Task-level recovery: Math rises to 66.49 after Code →Math but falls to 57.66 after later QA and CW training, while other domains remain largely stable.The pattern demonstrates domain-specific rather than uniform forgetting.
  • Task-level recovery: Math recovers from 57.66 to 66.04 after a short Re-Math refresh, approaching the Math-domain expert.The refresh recovers most of the loss from later-domain training.
  • Task-level recovery: Re-Math leaves the other domains nearly unchanged and achieves the best average score of 66.39.Code rises slightly to 51.05, while QA and CW remain essentially unchanged.
  • Direct intervention: The rollback selects MLP neurons using shared activation, QA-update magnitude, and directional conflict, then reverts only the selected QA increment.This intervention tests a coordinate proxy for the harmful component without identifying the latent conflict subspace.
  • Direct intervention: Reverting 2% of selected MLP neurons raises Math Avg from 59.90 to 61.25, recovering 20.4% of QA-induced Math loss while changing QA Avg by only −0.06.Random rollback of the same number of neurons instead lowers Math Avg to 59.49; a joint MLP+Attn selector reaches 73.6% recovery at a 32% budget.

7 Conclusion

The paper argues that cross-domain interference is localized to shared active routes rather than a full-model phenomenon, and that targeted recovery can preserve other domains.

  • Localized interference: Global gradients may be nearly orthogonal even though directional misalignment on shared active routes determines whether later training helps or harms earlier domains.RL updates are sparse, small, and have little overlap across domains, while reasoning domains reuse common routes.
  • Recovery: Theoretical and empirical results indicate that short refresh and sparse proxy rollback can restore damaged performance while largely preserving other domains.The paper identifies localized route-level interference control as a promising direction for more stable and scalable multi-domain RL.

Limitations

The paper identifies three limitations: no automatic targeted-refresh algorithm, a coarse conflict-subspace proxy, and uncertain transfer beyond multi-domain RL.

  • The proposed targeted-refresh route has not yet been developed into an automatic training algorithm.Future work includes automating degraded-domain detection, refresh order, and refresh budget.
  • The conflict-subspace intervention uses a coarse, potentially redundant coordinate proxy rather than a precise latent subspace estimate.The proxy is basis-aligned and omits normalization, residual-stream, and higher-order cross-module factors.
  • The analysis focuses on multi-domain RL, leaving its applicability to other post-training paradigms unresolved.The paper specifically identifies on-policy distillation as a setting for future extension.

A.1 Data Construction Details

The experiments use fixed-size domain datasets, held-out validation and test sets, shared GRPO training, and domain-specific reward mechanisms.

  • Data Construction Details: Each domain uses 5,120 training examples, with prompt lengths capped at 2,048 tokens.Validation sets contain 30 Math problems and 50 examples for each of Code, QA, and CW.
  • Data Construction Details: Creative-writing data combine four subsets, with 2,560 reference responses resampled using Qwen3-235B-A22B-Instruct-2507.
  • Data Construction Details: Code, QA, and writing evaluations use LiveCodeBench-v6, SuperGPQA, MMLU-Pro, and WritingBench-derived test procedures.The reported test sets contain 175 Code problems, 2,141 QA examples, and 1,203 MMLU-Pro examples.
  • Data Construction Details: All experiments use the same GRPO configuration, while checkpoints are selected by current-domain validation or average multi-domain performance.
  • Data Construction Details: Math and QA use binary answer-correctness rewards, Code uses execution-based rewards, and CW uses LLM-judged preference rewards.
  • Data Construction Details: Layer-wise analyses identify localized conflict and synergy in attention and MLP modules, with especially prominent conflicts for Math-QA.

B.2 Sequential Parameter Changes Remain Sparse

Sequential domain RL accumulates parameter changes across stages, but each individual stage remains a sparse incremental update rather than a global rewrite.

  • B.2 Sequential Parameter Changes Remain Sparse: 77.6% of parameters remain below the |∆W| < 10^-7 and r(W) < 10^-3 thresholds after Code, versus 73.4% after Code-Math-QA-CW.
  • B.2 Sequential Parameter Changes Remain Sparse: 87.2%, 84.0%, and 88.3% of parameters remain below the 10^-7 threshold during the Math, QA, and CW stages, respectively.
  • B.2 Sequential Parameter Changes Remain Sparse: Sequential training accumulates parameter shifts, while each stage still acts as a sparse incremental update.
  • B.2 Sequential Parameter Changes Remain Sparse: Weak overlap among edited neurons motivates analyzing shared active routes and directional alignment rather than direct co-editing alone.The framework measures active-route overlap and cosine alignment on shared edited neurons.
  • B.2 Sequential Parameter Changes Remain Sparse: The local perturbation analysis assumes smooth objectives, approximate stationarity, small effectively sparse updates, and a low-dimensional shared conflict subspace.Global near-orthogonality and positive curvature with weak cross-subspace coupling support the recovery analysis.
  • B.2 Sequential Parameter Changes Remain Sparse: At an approximately stationary checkpoint, later-domain damage is governed primarily by second-order curvature rather than first-order drift.Small sparse updates can still cause degradation when aligned with curvature-sensitive directions.

C.3 Proof of Proposition 2

Proposition 2 localizes the dominant second-order damage to the shared active conflict subspace, while the refresh analysis bounds collateral effects under near-orthogonality.

  • C.3 Proof of Proposition 2: The dominant second-order damage is the component of the later update projected into the low-dimensional shared active conflict subspace.Off-subspace leakage and purely off-subspace curvature contribute bounded residual terms.
  • C.3 Proof of Proposition 2: Low parameter overlap does not imply low interference because a small update can enter directions where the earlier objective is locally sensitive.
  • C.3 Proof of Proposition 2: Near-orthogonal global gradients can coexist with selective degradation because localized second-order displacement dominates inside shared active directions.
  • C.3 Proof of Proposition 2: Positive curvature and weak cross-subspace coupling make refresh contract the harmful component while treating off-subspace effects as controlled local errors.
  • C.3 Proof of Proposition 2: Alternating refresh follows, to first order, gradient descent on a weighted local objective and can approach a local Pareto-stationary compromise.This does not imply that all single-domain optima are simultaneously reachable.
  • C.3 Proof of Proposition 2: Re-Math improves rapidly before saturation while Code, QA, and CW remain largely stable, supporting a short local-correction regime.
  • C.3 Proof of Proposition 2: Prolonged Re-Code eventually makes Math decline more visible as accumulated displacement increases second-order damage.

D.4 Direct Rollback on a Coordinate Proxy for the Conflict Subspace

The study tests whether sparse, proxy-selected coordinate rollback can localize and reverse QA-induced Math damage without retraining. Recovery depends on conflict-aware selection and intervention budget, with joint MLP-attention rollback exposing additional recoverable damage beyond MLP coordinates.

  • Proxy rollback design: The intervention reverts only the QA increment on selected MLP gate/up/down parameters, requiring no additional optimization.The selected coordinates are chosen using a neuron-level proxy score and layer-wise budget allocation.
  • Proxy selector ablations: 20.4% of the QA-induced Math loss is recovered by the full selector, compared with 18.3% for the M × C selector.Removing update magnitude or directional conflict reduces recovery to about half, while A-only reaches 16.6% with recomputed layer allocation.
  • Proxy selector ablations: Conflict-aware selection produces a U-shaped layer budget concentrated in shallow and late layers, while M × C most closely matches the full selector’s support overlap.A-only can recover competitively under the full selector’s inherited budget despite smaller support overlap.
  • Intervention budget: Recovery rises from 2.0% at a 1% revert budget to 29.4% at 4%, then fluctuates around saturation rather than improving monotonically.The dose-response indicates that a small fraction of MLP neurons carries substantial harmful QA displacement, but MLP-only recovery remains incomplete.
  • Joint intervention: Attention intervention reverts corresponding WQ and WO slices, with shared KV rows reverted proportionally to the selected query heads.The joint intervention combines MLP neurons and attention units in one ranked list under a shared layer budget.
  • Joint intervention: Joint MLP+attention rollback recovers 36.1% at 4% and 73.6% at 32%, exceeding MLP-only recovery from 4% onward.At 2%, the joint selector recovers 12.7%, below the MLP-only 20.4%; recovery remains incomplete even at larger budgets.
Loading 2606.02398v1…