PaperlayerView exact revision on arXiv

Source-linked AI summary

$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu

arXiv:2609.37976v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Chain-of-thought can improve reasoning but carries high token costs, and the functional contribution of the Thinking–Non-thinking weight difference remains an open question. S³ selectively combines paired checkpoints by preserving the Non-thinking model’s leading spectral directions and importing the Thinking model’s complement. Across settings, it reduces token use while improving mean accuracy relative to full Thinking models.

  • Problem

    Chain-of-thought improves reasoning but increases decoding costs, and it remains unclear which part of the Thinking–Non-thinking weight difference can be removed without losing reasoning gains.

    Source: §1 · pp. 1–2

  • Method

    S³ is a training-free composition that preserves the Non-thinking model’s leading spectral directions and imports the complementary component from its paired Thinking checkpoint.

    Source: §1 · pp. 2–3

  • Results

    Across settings, S³ uses 27.4% fewer tokens on average than Thinking while improving mean accuracy by 1.0 percentage point.

    Source: §1 · p. 3

  • Takeaways & Limitations

    S³ achieves empirical Pareto-optimal accuracy–token trade-offs among evaluated training-free model-composition methods.

    Source: §1 · pp. 2–3

    The entropy-ordering analysis assumes mean attention entropy is locally minimized within the pretrained query-projection row space for a single attention head.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model's weight component within the null space of a projection defined by the corresponding Non-thinking model's dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap ($S^3$), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate $S^3$ on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical, multimodal, and audio reasoning domains. $S^3$ establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.

1 Introduction

The paper asks where Thinking checkpoints gain functional reasoning capability and whether that capability can be retained with less token overhead. It identifies the complementary spectral component as functionally important, proposes S³ to transfer it selectively, and reports accuracy–efficiency gains with an attention-entropy explanation.

Motivation and question

  • The paper asks which part of the Thinking–Non-thinking weight difference can be removed without losing reasoning capability or incurring full generation costs.It frames this against the computational cost of lengthy chains of thought and existing efficiency approaches.Source: §1 · pp. 1–2

Spectral finding

  • The aligned thinking-difference component contains substantial parameter energy but induces weak functional changes, while the complementary component drives most functional shifts.This motivates preserving the Non-thinking model in its dominant spectral subspace and importing the Thinking model’s complementary component.Source: §1 · pp. 2–3

Mechanistic explanation

  • Across evaluated datasets, attention entropy follows HNull < HBase < HSub, and a simplified analytical model explains when this ordering arises.The null-space model has more concentrated attention than Non-thinking models, while adding only the aligned component diffuses attention.Source: §1 · pp. 2–3

Selective composition

  • S³ is a training-free composition that preserves the Non-thinking model’s leading spectral directions and imports the Thinking model’s complementary component.Unlike whole-checkpoint interpolation or base-aligned update approaches, it assigns different spectral regions to the paired checkpoints.Source: §1 · pp. 2–3

Accuracy–efficiency results

  • Across settings, S³ uses 27.4% fewer tokens on average than Thinking while improving mean accuracy by 1.0 percentage point.It also achieves Pareto-optimal operating points, including an 8.3-percentage-point accuracy gain on HMMT25 with a 33.0% token reduction.Source: §1 · pp. 2–3

2 Related Work

The related-work discussion contrasts reasoning-efficiency methods and checkpoint composition with S³’s selective spectral transfer. It also connects attention entropy to reasoning dynamics and presents the paper’s parameter-space analysis as a theoretical contribution.

2.1 Efficient Reasoning and Chain-of-Thought Compression

  • S³ preserves the Non-thinking model in a protected spectral subspace and transfers only its paired Thinking model’s complement, without additional training or decoding changes.This contrasts with adaptive-computation and post-training approaches to reducing chain-of-thought overhead.Source: §2.1 · p. 3

2.2 Weight-Space Model Composition and Spectral Subspace Transfer

  • Unlike weight-space methods that average, interpolate, or merge updates, S³ uses the Non-thinking checkpoint as an anchor and the Thinking checkpoint as a selective reasoning-capability donor.The method transfers only the component outside the Non-thinking model’s leading spectral subspace.Source: §2.2 · p. 3

2.3 Attention Entropy and Reasoning Dynamics

  • The paper observes reduced attention entropy across models and datasets and analyzes how null-space projection in parameter space relates to attention entropy.Prior work is described as linking entropy to attention dynamics, reasoning trajectories, and reasoning efficiency.Source: §2.3 · p. 3

3 Method: Spectral Null-Space Swap

The method starts from the finding that functional differences between paired Thinking and Non-thinking checkpoints lie mainly outside the Non-thinking model’s occupied spectral directions. S³ protects selected leading directions and transfers the complementary component, with a ρ = 1 probe family that isolates the effects of the two components.

3 Method: Spectral Null-Space Swap

  • S³ turns the spectral finding into an operator that preserves leading Non-thinking directions while importing the Thinking model’s remainder, controlled by ρ.The method’s three steps are to locate the functional difference, define the swap, and examine the full-protection probe family.Source: §3 · p. 4

3.1 Insight: Where the Thinking Difference Lives

  • The thinking difference has substantial energy inside the Non-thinking model’s spectral subspace but produces much less functional change there than in its complement.The split compares weight-space energy with hidden-state changes measured from model computation.Source: §3.1 · p. 4

3.2 The Spectral Null-Space Swap

  • For a protected subspace Sρ, the swap keeps the Non-thinking weights inside it and takes the Thinking weights on its orthogonal complement.Equivalently, it discards the aligned part of the thinking difference and transfers only the component outside Sρ.Source: §3.2 · pp. 4–5

3.3 Specialization at ρ = 1: The Probe Family

  • At ρ = 1, the swap transfers the component of the thinking difference that the Non-thinking weight cannot represent.For MLP matrices with m > n, the relevant component is the left-null-space component, making the expression well-defined for every target weight.Source: §3.3 · pp. 5–6
  • The ρ = 1 probe family comprises Base and Full checkpoints plus derived Sub and Null variants, all available from the paired checkpoints without training, data, or roll-outs.The four variants differ in which spectral components of the same weight difference are applied, supporting attribution to particular directions and layers.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate paired Qwen3 checkpoints across text, vision–language, and audio reasoning tasks, comparing S³ with the original checkpoints and weight-merging baselines. The reported results favor S³’s accuracy–token trade-off, while ablations examine the protected-subspace ratio and the distinct contributions of the subspace and null-space components.

4.1 Experimental Setup

  • The evaluation uses paired Qwen3 dense and MoE checkpoints across text, vision–language, and audio tasks, with pass@1 and avg@4 metrics.Baselines include the Non-thinking and Thinking checkpoints, MI-0.8, and TIES-Merging, evaluated with identical prompts and decoding configurations.Source: §4.1 · p. 6, §4 · p. 6

4.2 Main Results: Cross-Modal Accuracy–Efficiency Trade-offs

  • S³ provides a favorable accuracy–efficiency trade-off across text, vision–language, and audio reasoning, with a superior Pareto frontier to TIES and MI-0.8.The main-results table reports accuracy and average generated tokens per problem; the prose reports recovery of most Thinking gains with shorter outputs.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Ablation on the Protected-Subspace Ratio ρ

  • Smaller ρ generally improves reasoning capability but lengthens generations, while larger ρ yields shorter outputs; ρ = 0.8 is chosen as a balance.The ratio ablation varies ρ from 0.5 to 1.0 for Qwen3-4B-S3.Source: §4.3 · pp. 6–7, Table 2 · p. 8

4.4 Ablation on Subspace and Null-Space Components

  • Across math benchmarks, the Sub model stays close to Base, while Null matches Full, indicating that the reasoning gain is concentrated in the null-space component.Table 3 compares accuracy and token length across five math benchmarks.Source: §4.4 · p. 8, Table 3 · p. 8

5 Mechanistic Analysis

Across tested datasets, null-space variants exhibit more concentrated attention than baseline and subspace variants, alongside improved reasoning efficiency. A simplified local model explains this ordering through different entropy effects of active-subspace and null-space perturbations.

5.1 Preliminary: Attention Entropy and Reasoning Efficiency

  • Across tested datasets, HNull < HBase < HSub, with null-space showing more concentrated attention and improved reasoning efficiency.The ordering also holds at the per-problem level.Source: §5.1 · pp. 8–9
  • The Sub variant has the highest entropy, longest reasoning traces, and highest SimpleQA hallucination rate; prior work links diffuse attention to higher cognitive load [50] and longer CoT [55] [56].Source: §5.1 · p. 8

5.2 A Simplified Analytical Model for the Subspace Entropy Ordering

  • The analysis assumes entropy is locally minimized within the pretrained query-projection row space, motivated by training-induced stationarity there.The orthogonal null space is treated as retaining nontrivial first-order variation.Source: §5.2 · p. 9
  • Under that assumption, only null-space perturbations contribute at first order, while active-subspace perturbations cannot locally decrease entropy.This accounts for the higher entropy of Sub than Base.Source: §5.2 · p. 9
  • In the simplified model, null-space projection isolates entropy reduction, subspace projection isolates entropy increase, and Full interpolates between them.Source: §5.2 · p. 10

6 Conclusion

S3 combines paired Non-thinking and Thinking checkpoints by retaining the former's dominant spectral subspace and importing the latter's complementary components. Across more than 25 settings, it reduces generated tokens while maintaining or improving accuracy; the paper connects this efficiency pattern to lower attention entropy.

6 Conclusion

  • Approximately 30% fewer generated tokens while maintaining or improving accuracy over Thinking baselines across more than 25 evaluation settings.The conclusion covers 2B–30B dense and MoE models and text, vision-language, and audio reasoning benchmarks.Source: §6 · p. 10
  • S3 preserves the Non-thinking model inside its primary spectral subspace and imports only complementary components from the paired Thinking model.The complementary null-space component is described as driving core functional shifts, while the aligned component has substantial parameter energy but weak functional impact.Source: §6 · p. 10
  • Null-space projection produces lower attention entropy and more concentrated attention, which the paper explains through local stationarity.Source: §6 · p. 10

A.1 Layer Selection

  • The analysis measures attention entropy in layers 32–35, the final four layers of Qwen3-4B, because prior work links late-layer attention states to reasoning-critical computation.Source: §A.1 · p. 15

A.2 Full Dataset Attention Entropy Results

The appendix reports that the entropy ordering among Null, Base, and Sub holds across seven datasets, and relates entropy to SimpleQA hallucination and pretraining dynamics. It also finds that nearly all measured entropy reduction occurs early in pretraining.

A.2 Full Dataset Attention Entropy Results

  • HNull < HBase < HSub on every one of the seven datasets, with Full between Null and Sub throughout.The reported metric is mean attention entropy averaged over layers 32–35.Source: Figure 4 · p. 15, §A.2 · p. 15

A.3 Hallucination and Attention Entropy

A.4 Entropy Saturation During Pretraining

  • The entire attention-entropy reduction occurs within approximately the first 2% of pretraining steps.The authors interpret the remaining regime as near-stationary in the determined subspaces.Source: Figure 5 · p. 16, §A.4 · p. 16

A.5 Proofs

The proofs model attention entropy as a smooth function of a query projection and decompose perturbations into orthogonal active-subspace and null-space components. The appendix also describes the model suite and rollout protocol used for additional experiments.

A.5 Proofs

  • The proof decomposes the entropy gradient and weight perturbation into orthogonal row-space and null-space components under the Frobenius inner product.The active subspace is V = row(WQ,0), with V⊥ its orthogonal complement.Source: §A.5 · p. 16
  • Under local optimality, active-subspace perturbations have zero first derivative and cannot locally decrease entropy for sufficiently small perturbations.The measured relative perturbation magnitude is approximately 8%.Source: §A.5 · p. 17
  • Across seven datasets, the null-space component has a consistently negative inner product with the entropy gradient, supporting a first-order entropy decrease for sufficiently small perturbations.Source: §A.5 · p. 17, Figure 6 · p. 17

B.1 Additional Experiment Configuration

  • Additional experiments pair Qwen3 dense, MoE, vision-language, and audio models; each example receives four seeded rollouts, with pass@1 and avg@4 reported.Source: §B.1 · p. 18

B.2 Results on Additional Benchmarks and Pareto-Frontier Analysis

Additional benchmarks show S3 retaining much of Thinking’s reasoning performance with fewer generated tokens. Across text, vision–language, and audio settings, it often occupies a favorable accuracy–token trade-off, including operating points complementary to TIES and MI-0.8.

Additional LLM benchmarks

  • On additional math benchmarks, S3 retains Thinking’s accuracy gains particularly well on AIME while substantially reducing generated tokens at both model scales.It remains competitive with Thinking and alternative merging baselines on AMC23 and MATH-500.Source: §B.2 · p. 18, Table 8 · p. 19
  • Table 8 reports accuracy and average generated tokens per problem for the additional math benchmarks.The table’s bolding compares TIES, MI-0.8, and S3 within each column.Source: Table 8 · p. 19

Pareto-frontier analysis

  • Across modalities, S3-0.8 frequently reduces output length while retaining competitive reasoning performance, recovering substantial Thinking capability without its full chain-of-thought overhead.Figures 7–9 show the accuracy–token trade-off across the evaluated benchmarks.Source: §B.2 · pp. 18–19
  • Compared with TIES and MI-0.8, S3-0.8 often uses fewer tokens at comparable performance, although those baselines achieve stronger accuracy on some benchmarks.Similar accuracy–efficiency patterns appear in vision–language and audio reasoning as well as text-only reasoning.Source: §B.2 · pp. 18–19

B.3 Additional Ratio-Sweep Results.

The protected-subspace ratio ρ controls how much of the Thinking model’s complementary component is transferred, creating different accuracy–token operating points. Its useful setting varies by task, while the sweeps show that the control extends across dense, MoE, and multimodal models.

B.3 Additional Ratio-Sweep Results.

  • The ratio sweeps extend the control across dense, MoE, and vision–language model families, with Figures 7–9 visualizing accuracy–token frontiers across LLM, vision–language, and Omni benchmarks.The figures use paired model-family markers for Qwen3-4B and Qwen3-30B-A3B in Figure 7, and Qwen3-VL-2B and Qwen3-VL-4B in Figure 8.Source: §B.3 · p. 19, Figure 7 · p. 20, Figure 8 · p. 20
  • On text models, lower ρ often improves challenging AIME25 performance but generally produces longer generations, while larger ratios frequently minimize token use.The best ratio varies by benchmark, with some favoring more transfer and others an intermediate setting.Source: §B.3 · p. 20, Table 9 · p. 21
  • Vision–language benchmarks also favor different ratios: lower settings can help some tasks, while intermediate or higher settings perform better on others.For Qwen3-VL-4B, ρ = 0.8 is strongest on MMMU and MathVista-testmini.Source: §B.3 · p. 20, Table 9 · p. 21
  • Across model families, ρ provides a post-hoc way to select an accuracy–cost operating point, and the authors use ρ = 0.8 as a robust default balance.The setting requires neither retraining nor a change to the decoding procedure.Source: §B.3 · pp. 19, 21

C.1 Mathematical Reasoning

The mathematical examples compare how S3 and baselines handle problems where the arithmetic is often shared but the derivations differ. Across examples, S3’s traces are shorter, sometimes alongside a correct answer when baselines err.

Example set

  • Across nine examples spanning algebra, number theory, counting, and geometry, seven include at least one baseline that also answers correctly, allowing style comparisons beyond accuracy alone.In the April 1 calendar example, all four models agree on the mathematics, while their derivations and token use differ.Source: §C.1 · p. 22

Example 1: When April 1st Is a Sunday Again

  • 69.8% fewer tokens: S3 and Thinking both correctly identify 2018 as the next year April 1 falls on a Sunday, while TIES and MI-0.8 give different years.S3 states the weekday-shift rule once and avoids the extra derivations, verification, and recap described for the longer traces.Source: §C.1 · p. 22

Example 2: Volume Enclosed by Three Distances

  • 41.7% fewer tokens: S3 and TIES compute the correct volume, 288π, while Thinking and MI-0.8 return descriptions of the solid instead.The models share the steps identifying a sphere and solid ball; the divergence is whether to compute the requested volume.Source: §C.1 · p. 22

Example 3: A Product of Fractions Equal to 51/2010

  • 71.6% fewer tokens: S3 and MI-0.8 construct a valid set of size 39, while Thinking and TIES search numerically and conclude the minimum is 40.The correct traces use a substitution to reach the target; the incorrect traces treat a failed search as evidence that no such set exists.Source: §C.1 · pp. 22–23

Additional examples

  • Other examples contrast S3’s targeted solution steps with baseline approaches: recurrence addition, interpolation, multiplicative-chain decomposition, and bounding a moving-average term.S3 also uses a polynomial substitution before Lucas in the trinomial example and reduces the rational-squares problem to how many squares 7 needs.Source: Table 10 · p. 23, Table 14 · p. 27, Table 15 · p. 28

C.2 Vision Language Reasoning

Across six figure-based examples, S3 often answers correctly where baselines fail, while using substantially fewer tokens. The examples show savings from reading a visual relation once rather than repeatedly revisiting it.

C.2 Vision Language Reasoning

  • In four of six figure-based examples, no baseline answers correctly; in the other two, exactly one does.Source: §C.2 · p. 32
  • 62.7% fewer tokens: S3 counts one tandem bike behind the brown bicycle and one matte truck, while all three baselines miss the spatial relation.The baselines convert left/right position into front/behind; the image does not support that inference.Source: §C.2 · p. 32, Table 22 · p. 36
  • 75.6% fewer tokens: S3 identifies the blank as company D’s expenses and makes one substitution, while all three baselines lose track of the requested cell.S3 obtains 1,081,584 and option (C).Source: §C.2 · pp. 32–33, Table 23 · p. 37
  • 87.1% fewer tokens on the food-web example and 95.5% fewer on the loanable-funds diagram; S3 follows the relevant edge or curve change, unlike the baselines’ errors.Source: Table 19 · p. 33, Table 24 · p. 38
  • 81.5% fewer tokens: S3 reads the clock hands once and keeps the correct 4:30 reading, while baselines report 5:30.All four traces inspect the same pixels; the reduction reflects the other models reopening their reading.Source: §C.2 · p. 32, Table 20 · p. 34
  • 67.3% fewer tokens: S3 subtracts the two angle labels at A, while Thinking and MI argue over the vertex order in “ABDC.”Source: Table 21 · p. 35

C.3 Audio Reasoning

The audio examples show S3 using decisive spoken cues that baselines miss, with shorter responses; in four of five examples, no baseline answers correctly.

C.3 Audio Reasoning

  • In four of five audio examples, no baseline answers correctly; the fifth is included as a contrast where all four methods succeed.The examples cover listening comprehension, MMAR audio reasoning, and MMSU disfluency annotation.Source: §C.3 · p. 39
  • 55.0% token reduction: S3 identifies the fourth speaker’s admission as the answer, while all three baselines infer the culprit from other cues and choose incorrectly.Thinking and MI-0.8 infer guilt from a cough; TIES infers guilt from ownership language.Source: §C.3 · p. 39
  • 70.9% token reduction: S3 counts the four individually demonstrated Mandarin tones, while all three baselines answer three by counting only the six later syllables.MI-0.8 explicitly notes the earlier demonstrations before ruling out the fourth tone.Source: §C.3 · p. 39
  • 41.8% token reduction: S3 classifies the opening laugh as a chuckle rather than a standard filled pause and answers correctly, while all three baselines choose incorrectly.The other disfluencies are a discourse marker and a restart.Source: §C.3 · pp. 39–40
  • Two further audio cases show S3 using “test” as a domain cue where baselines supply a nonexistent fitness practice, and recalling trifle layers without rechecking each ingredient.The trifle baselines re-check distractor ingredients one by one.Source: Table 25 · p. 40, Table 27 · p. 42

Related papers

Loading 2609.37976v2…