PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Rewiring for Exploration, Purification, and Model Merging

Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

arXiv:2607.03065v1cs.LGcs.AI

TL;DR

Dense full-parameter reinforcement-learning updates can suppress reasoning and cause cross-domain interference, limiting control over post-training changes. This paper introduces Subspace-Aligned Rewiring, which projects updates into the base model’s spectral manifold; across extraction, purification, and merging, it preserves or improves downstream performance using compact updates.

  • Problem

    Full-parameter reinforcement-learning updates offer limited control and can suppress reasoning performance or induce cross-domain interference during capability consolidation.

  • Method

    Subspace-Aligned Rewiring projects dense reinforcement-learning updates onto the pretrained model’s spectral manifold to retain reasoning-effective structure and filter residual directions.

  • Results

    SAR preserves or improves downstream performance across extraction, mixed-domain purification, and expert merging, using as little as ∼0.58% of total parameters for reasoning extraction.

  • Takeaways & Limitations

    Spectral rewiring serves as a training-free, post-hoc tool for improving reasoning and multi-domain model performance.

  • Takeaways & Limitations

    SAR’s effectiveness is bounded by projection compatibility, namely how much of an update can be represented as reorganizing knowledge already encoded in the reference model.

Abstract

from arXiv · show

Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.

1 Introduction

The paper introduces Subspace-Aligned Rewiring (SAR), which projects dense reinforcement-learning updates onto the pretrained model’s spectral manifold to preserve reasoning-effective components while removing residual directions. This post-hoc editing enables compact extraction, improved exploration, cross-domain purification, and expert merging.

  • Motivation: Full-parameter reinforcement learning can suppress reasoning performance through narrow high-reward trajectories and create interference among mathematical, coding, and instruction-following capabilities.These bottlenecks motivate separating reasoning-effective update components from residual parameter-space directions.
  • Method: SAR projects dense reasoning updates onto the pretrained model’s SVD coordinates, preserving the base model’s spectral basis while removing off-manifold components.The method treats singular vectors as pretrained atomic skills and singular values as their original connections.
  • Extraction: ∼0.58% of total parameters preserves more than 99% of peak Pass@1 across 1.5B–32B reasoning models, while top-1% projection improves six of seven open agentic coding benchmarks.The in-house agentic coding improvement is +2.52% on average.
  • Exploration: SAR surpasses unconstrained full-parameter reinforcement learning in Pass@k scaling, crossing the full-RL curve at small k and expanding the set of AIME problems reachable through repeated sampling.The result indicates improved high-k exploration beyond the base model.
  • Purification and merging: On OLMo-3.1-32B-Think, SAR improves LiveCodeBench v5 by +1.48% and v6 by +0.95%, raises AIME 25 Pass@16 from 88.33% to 91.71%, and maintains instruction following.In expert merging, SAR surpasses the best single-domain experts at both 1.5B and 14B scales while improving AIME and LiveCodeBench simultaneously.

2 The Geometric Framework of Reasoning Elicitation

The section develops a geometric framework in which the pretrained model’s spectral subspace serves as a functional basis for reasoning, and introduces SAR to extract reasoning-effective updates within that space. It further interprets off-diagonal spectral rewiring as enabling multi-conditional deductions and provides a compact interface for compression, filtering, and merging.

  • Subspace-Aligned Rewiring: SAR projects full-parameter RL updates onto the base model’s spectral subspace, retaining the reasoning-effective component while discarding the orthogonal residual.It first extracts the top-k low-rank update and then reconstructs its projection using the pretrained SVD basis.
  • Spectral basis: The pretrained model’s principal spectral subspace provides a compact coordinate system for its functional library and reasoning updates.Singular vectors encode input signals and output representations, while the model’s functional capacity is described as concentrated in this subspace.
  • Mechanism of reasoning: Off-diagonal rewiring entries connect distinct latent input features to output directions, enabling many-to-one logical synthesis and multi-conditional deductions.Diagonal entries rescale isolated capabilities, whereas off-diagonal entries integrate evidence across spectral components.
  • Downstream operations: The rewiring matrix turns dense weight changes into an editable spectral object, allowing compression, filtering, and merging to operate on the rewiring matrix instead of the full update.This perspective treats post-training as changing connections among pretrained skill directions rather than replacing the learned basis.

3 Extracting the Reasoning Core: Single-Domain Validation

SAR extracts compact spectral updates that preserve full-RL mathematical-reasoning gains across model scales while improving high-k exploration. The same post-hoc editing approach also improves applied agentic coding performance without retraining.

  • Mathematical reasoning: SAR-projected models match full-RL mathematical reasoning within a small AVG@32 margin while often matching or slightly improving Pass@32.The results support concentrating evaluated RL-induced reasoning improvements in the pretrained spectral subspace.
  • Compact reasoning core: Retaining only 1% to 10% of pretrained spectral rank recovers full-RL-level reasoning performance for the evaluated models.The corresponding rewiring matrix is much smaller than the full model, indicating that much of the unconstrained RL update is unnecessary for measured AIME gains.
  • High-k exploration: SAR crosses full-RL Pass@k at k = 2 for DeepScaleR and k = 4 for OLMo-3.1-32B-Think, then remains better at larger k.With 256 AIME 2024 rollouts, SAR covers 26/30 problems versus 25/30 for full RL.
  • Ablations and mechanism: No-projection, random, diagonal-only, and off-diagonal-only controls perform worse than SAR, indicating that its gains are not explained by low-rank extraction alone.The results characterize SAR as retaining reasoning-effective spectral directions while removing orthogonal residual updates.
  • Agentic coding: +2.52% average improvement and gains up to 25.10% result from a top-1% spectral-rank projection across six of seven open agentic coding benchmarks.This validation uses the same coding post-training delta on an in-house model.

4 Cross-Domain Generalization via Spectral Reasoning Rewiring

SAR projects reasoning updates onto the pretrained spectral manifold to reduce cross-domain interference while retaining shared reasoning structure. In jointly trained and merged models, it improves coding and exploration while preserving core math and instruction-following performance.

  • Mix-RL purification: SAR projects reasoning updates onto the base spectral manifold, filtering domain-specific residual directions that interfere across mathematics, coding, and instruction following.The method uses the pretrained spectral geometry to retain shared rewiring components and remove less compatible task-specific directions.
  • Mix-RL purification: 69.09% on LCB v5 and 65.25% on v6 are achieved by SAR, up from 67.61% and 64.30% for the full RL baseline.Pass@k also improves consistently on both LiveCodeBench versions.
  • Mix-RL purification: 91.71% on AIME 25 Pass@16 is reached by SAR, up from 88.33%, while peak math performance varies by less than 1% in AVG@32.IFEval changes only slightly from 92.42% to 92.05%, indicating retained instruction-following capacity.
  • Mix-RL purification: Less than 1% of model parameters in the rewiring matrix M suffices to improve LiveCodeBench and recover larger-budget math and coding exploration capacity.SAR preserves RL accuracy gains while exposing a compact spectral rewiring more compatible across domains.
  • Expert merging: At 1.5B and 14B scales, SAR-merged models surpass the best single-domain experts on primary math and code AVG metrics.At 1.5B, AIME AVG@32 reaches 43.44% and LCB AVG@8 reaches 32.25%; at 14B, they reach 74.38% and 63.40%, respectively.

5 Discussion

SAR preserves or improves downstream performance by extracting reasoning-effective components of dense RL updates in the pretrained model’s spectral coordinates. Its effectiveness depends on whether the reference model already contains the relevant knowledge and whether RL primarily reorganizes it.

  • Central insight: SAR preserves or improves key downstream performance across single-domain extraction, Mix-RL purification, and expert merging by projecting dense RL updates into pretrained spectral coordinates.The results support viewing reasoning-effective RL changes as compact rewiring of capabilities already represented in the reference model’s spectral geometry.
  • Expert merging: SAR outperforms naive interpolation and improves cross-domain transfer when compact reasoning-effective parameters are extracted and integrated through model merging.Table 4 characterizes this as cross-domain generalization via model merging.
  • Scope: SAR’s scope assumes the reference model already contains the relevant reasoning ability or domain knowledge and RL primarily elicits or reorganizes it.The method’s central premise is that useful adaptation can be represented as a reorganization of existing capability.
  • Limitations: SAR’s main boundary is projection compatibility: how much of an RL update can be represented as reorganizing knowledge encoded in the reference model.This boundary concerns the match between the update and the reference model’s existing spectral representation.
  • Limitations: After more than 4000 training steps, projecting JustRL still recovers an AIME 2024 score of about 43%, comparable to DeepScaleR-level reasoning, but below the final RL model.The result indicates that spectral projection captures substantial reasoning content without fully reproducing the heavily optimized RL update.

6 Conclusion · Appendix · A Related Work

The paper presents a geometric account of RL post-training in pretrained spectral coordinates and introduces SAR as a post-hoc method for extracting reasoning-effective updates. It concludes that SAR preserves reasoning, improves exploration, reduces multi-domain interference, and strengthens expert model merging.

  • 6 Conclusion: The paper develops a first-principles geometric view of RL post-training in large language models.
  • 6 Conclusion: SAR captures reasoning-effective change through compact spectral rewiring aligned with pretrained singular-vector coordinates.The authors relate this alignment to RL reorganizing latent capabilities already represented in the base model.
  • 6 Conclusion: SAR preserves full-RL-level reasoning, improves high-k exploration, reduces multi-domain interference, and strengthens expert model merging.
  • A Related Work: Prior exploration work treats the exploration–exploitation trade-off mainly as a training-time RL objective-design problem.Related approaches target Pass@k-style behavior or test-time scaling efficiency.
  • A Related Work: Prior adaptation and RL-dynamics studies use low dimensionality as a training parameterization, compression principle, or predictability property.SAR instead identifies alignment between the reasoning-effective full RL update and pretrained spectral coordinates.
  • A Related Work: Model-merging methods combine task deltas with arithmetic, magnitude, sign, or sparsity-based rules to reduce interference.Related analysis finds RL-trained models more mergeable than SFT-trained models, although merged models can still degrade relative to corresponding experts.

B Illustrative Example of Spectral Rewiring

The example shows how SAR extracts reasoning-relevant off-diagonal spectral components for triangle-area calculation. These components connect side-length and angle information to the output direction implementing the sine-area formula.

  • The illustrative task computes a triangle’s area from two side lengths and their included angle.
  • In the pretrained model, side-length and angle capabilities occupy separate spectral directions, while an output direction represents applying the sine-area formula.
  • Because the original diagonal mapping cannot activate the output direction from either input direction alone, SAR isolates the RL-induced off-diagonal entries M3,1 and M3,2.

C Implementation Details

SAR is applied to attention and MLP linear weights while preserving biases, normalization parameters, embeddings, and the language-modeling head from the base model. Evaluations use shared sampling settings, FP32 decompositions, and precision choices intended to preserve the projected spectral geometry.

  • Algorithmic Scope: SAR projects only the attention and MLP linear weights in each Transformer block, leaving biases, normalization parameters, embeddings, and the language-modeling head unchanged.The projected weights are Wq, Wk, Wv, Wo, Wgate, Wup, and Wdown.
  • Algorithmic Scope: Projecting both attention and MLP weights is necessary to recover the full effect, whereas projecting either alone degrades performance relative to full RL.Using base or RL versions of embeddings and the language-modeling head has little effect on final results.
  • Evaluation Configuration: All reasoning and coding evaluations use temperature T = 0.6, top-p = 0.95, and a maximum generation length of 32,768 tokens unless otherwise specified.POLARIS uses a maximum generation length of 96,000 tokens; compared models within each experiment share identical sampling parameters.
  • Numerical Precision: SVD decompositions and SAR projections are computed in FP32, while projected models are stored in FP16 to better preserve fine-grained updates than BF16.The stated storage choices apply to the 1.5B, 4B, 7B, and 32B OLMo-3 models, with the latter evaluated in BF16.
  • Coverage Analysis: SAR solves one additional AIME 2024 problem beyond full RL under 256-rollout coverage, expanding the set of reachable correct solutions rather than merely changing single-sample accuracy.A problem is covered when any rollout produces the correct final answer.

D Boundary Cases of Spectral Rewiring

SAR is most effective when sparse-reward RL makes small updates that reorganize capabilities already encoded in the reference model. When training is intensive or domain knowledge is missing, spectral projection remains informative but provides an incomplete reconstruction.

  • Scope assumption: SAR assumes an elicitation regime in which sparse verifiable rewards and token-averaged updates make relatively small per-step changes.These updates reorganize capabilities already encoded in the reference model.
  • Heavily trained reasoning RL: Over 4000 RL steps in JustRL reduce reconstruction completeness: base-spectral projection recovers roughly 43% AIME 2024, versus ∼52% for the final RL model.The projected score remains comparable to the earlier DeepScaleR-level result.
  • Direct code RL from a base model: Direct code RL can exceed the base model’s spectral geometry because programming performance requires syntax, execution protocols, exact formatting, tests, and edge-case handling.These interface and programming behaviors may not be sufficiently established in the base model.
  • Dense critic rewards and PPO-style training: Dense token-level rewards and critic-shaped PPO objectives can produce larger, more continuous parameter movement, reducing compatibility with reference spectral coordinates.The limiting factor is effective optimization strength rather than the PPO algorithm itself.
  • Scope of SAR: SAR captures RL aligned with the reference spectral manifold, but missing domain knowledge or stronger training can force behavior outside that manifold and yield incomplete reconstruction.Projection remains informative when the reference model already contains the relevant reasoning ability or domain knowledge.

E Comparison with the No-Projection Control

The no-projection control shows that low-rank extraction alone preserves some single-sample reasoning and code-side gains, but spectral alignment is necessary for stronger exploration, math reasoning, and cross-domain transfer.

  • Control Design: SAR and the no-projection control use the same rank budget; only SAR projects the extracted update onto the base model’s pretrained SVD subspace.This isolates spectral alignment from low-rank extraction.
  • Reasoning and Exploration: At Pass@256, SAR solves one additional AIME 2024 problem that neither full RL nor the no-projection control solves within 256 attempts.The control matches SAR at Pass@1 but saturates earlier as k increases.
  • Multi-Domain RL: 93.33% Pass@32 is achieved by SAR on AIME 2025, versus 86.67% by the no-projection control.The control retains part of AIME performance but fails to recover SAR’s exploration benefit.
  • Cross-Domain Merging: The no-projection math-to-code merge preserves code-side gains but substantially weakens math reasoning compared with SAR.Both methods merge the same 1% low-rank component from the math RL update, but only SAR aligns it with the pretrained spectral basis.
  • Overall Conclusion: The comparison indicates that reasoning-effective parameters are compact and specifically expressed in the base model’s pretrained SVD space.Low-rank extraction explains part of parameter efficiency, while spectral alignment yields stronger exploration and cross-domain transfer.

F Method Ablations

Method ablations show that SAR’s pretrained spectral alignment and full rewiring structure are necessary for reliable reasoning gains. Random projection improves over the base model but trails SAR, while diagonal-only and off-diagonal-only controls fail to reproduce the full benefit.

  • Rewiring structure: The off-diagonal-only variant is included as a complementary partial-rewiring control, alongside diagonal-only rewiring and random projection.The supplied results passage is truncated before stating the off-diagonal-only outcome.
  • Projection basis: Random projection improves Pass@1 over the base model but remains clearly below SAR, showing pretrained singular-vector coordinates are not interchangeable with arbitrary low-dimensional projections.The comparison replaces the pretrained spectral basis with a random projection under the same spectral budget.
  • Rewiring structure: The diagonal-only variant fails to recover the RL gain, indicating that rescaling isolated pretrained components is insufficient for reasoning elicitation.This control retains only the diagonal entries of the rewiring matrix M.
  • Ablation setup and results: On DeepScaleR-1.5B with a 1% AIME 2024 update budget, SAR preserves full-RL-level Pass@1 while maintaining Pass@32.Random projection and partial rewiring controls lose reliability or coverage.
Loading 2607.03065v1…