PaperlayerView exact revision on arXiv

Source-linked AI summary

S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models

Jack Young

arXiv:2604.01168v2cs.CLcs.LG

TL;DR

Hybrid models expose recurrent initial states that existing PEFT methods leave at zero, raising whether this state can support effective adaptation with scarce verified supervision. S0 tuning learns one initial state matrix per recurrent layer while freezing weights and achieves strong HumanEval gains with zero inference overhead, alongside significant but narrower cross-domain transfer. Its scope is bounded by the need for execution-verified solutions and weaker or absent transfer on some tasks.

  • Problem

    Existing LoRA and prefix tuning target weight matrices while leaving hybrid models’ recurrent states at zero, limiting evidence about this adaptation surface under scarce verified supervision.

  • Method

    S0 tuning optimizes one initial recurrent state matrix per layer with all model weights frozen, adding zero inference overhead.

  • Results

    +23.6 ± 1.7 pp greedy pass@1 on Qwen3.5-4B, with S0 outperforming LoRA by +10.8 pp (p < 0.001); transfer is smaller on MATH-500 and GSM8K and absent on Spider.

  • Takeaways & Limitations

    Recurrent state initialization is a competitive zero-inference-overhead PEFT surface for hybrid language models in the tested small-data regime.

  • Takeaways & Limitations

    S0 requires execution-verified correct solutions, and the headline evidence comes from a small training set; applying it where verification is expensive remains untested.

Abstract

from arXiv · show

Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.

1 Introduction

S0 tuning adapts hybrid recurrent-attention models by learning their initial recurrent states while freezing model weights, yielding strong small-data HumanEval gains and competitive performance across two architectures. Mechanistic evidence indicates that this initialization steers generation trajectories from the first output character.

  • Method: S0 tuning learns one initial recurrent state matrix per layer with frozen weights, targeting an adaptation surface left unused by LoRA and prefix tuning.Hybrid models retain recurrent state matrices that accumulate context information and are initialized to zero by default.
  • HumanEval results: +23.6 ± 1.7 pp greedy pass@1 on Qwen3.5-4B, with S0 outperforming LoRA by +10.8 pp (p < 0.001) across 10 seeds.The Qwen3.5-4B model uses GatedDeltaNet recurrence.
  • Controls: −13.9 pp under all nine configurations for prefix tuning on pure-Transformer Qwen2.5-3B, contrasting with S0’s gains on hybrid models.The result serves as a negative control for applying prefix tuning outside recurrent-attention hybrids.
  • Mechanistic interpretation: 23 of 27 FAIL-to-PASS flips, or 85%, diverge from baseline at the first generated character, supporting trajectory steering by the initialized recurrent state.S0’s direct influence on output logits decays to 0.03% KL by the end of the prompt, while the generation trajectory has already diverged.
  • Scaling and variants: +27.1 pp for a per-step state-offset variant, exceeding S0 and LoRA but adding per-step inference cost.The standard S0 method retains zero inference overhead; the offset variant does not.

2 Background

Hybrid recurrent-attention models update full or structured recurrent states during sequence processing, creating an adaptation surface whose expressiveness depends on the state representation. GatedDeltaNet and Mamba-2 use related update mechanisms, while diagonal states lack cross-feature interactions.

  • GatedDeltaNet: Each GatedDeltaNet recurrent layer maintains a state matrix St updated through a gated delta rule.The state is described as a matrix updated at every token.
  • GatedDeltaNet: The gated delta rule uses αt as a decay gate and βt as write strength, with kt and vt serving as key and value vectors.The update erases the old association for key kt before writing the new one.
  • Mamba-2: Mamba-2 uses a structurally similar recurrent update through its structured state-space duality, with a scalar gate and input projection.The recurrence is presented as an alternative implementation of state evolution in a hybrid architecture.
  • State representation: A GatedDeltaNet layer in Qwen3.5-4B carries approximately 524K state entries in a full matrix representation.The cited dimensions are H=32 and K=V=128.
  • State representation: Mamba-1 instead uses a diagonal state with 16 independently evolving dimensions and no cross-feature interactions, unlike the full matrix states discussed for hybrid models.The text links this structural gap to the potential expressiveness of initial-state tuning.

3 Method

S0 tuning learns one initial recurrent-state tensor per recurrent layer while freezing backbone weights, injecting it before the first token. The state is absorbed after t=1, giving zero inference overhead without weight merging.

  • S0 tuning introduces a learnable tensor matching each recurrent layer’s native state shape while freezing all backbone parameters.The recurrent layer starts from αS0 instead of the default zero state.
  • S0 is initialized per recurrent layer and injected as the hidden state before the first token.
  • S0 is optimized by gradient descent on a completion-only loss over correct solutions, with all model weights frozen.Prompt tokens are masked so gradients act only on the target completion.
  • S0 is absorbed into the recurrent state at t = 1, so subsequent inference uses the unmodified recurrence with no adapter branch or weight merging.
  • The state-scaling hyperparameter is architecture-specific: α = 0.07 for Qwen3.5 and α = 0.65 for FalconH1.
  • The state-offset variant adds a learned tensor at every timestep and therefore incurs non-zero inference overhead.

4 Experimental Setup

The experiments evaluate S0 across two hybrid architectures, held-out HumanEval, cross-domain benchmarks, and LoRA and prefix-tuning controls. The design emphasizes a 10-seed Qwen comparison while treating 3-seed Falcon results as supportive evidence.

  • Qwen3.5-4B interleaves 24 GatedDeltaNet and 8 attention layers, while FalconH1-7B runs Mamba-2 and attention heads in parallel.Their recurrent-state parameter totals are 12.6M and 34.6M, respectively.
  • HumanEval uses 164 problems, with problems 80–163 held out and roughly 48 execution-verified training solutions collected from the frozen base model.
  • The primary baseline is LoRA, while matched-parameter prefix tuning on pure-Transformer Qwen2.5-3B serves as a negative control.
  • Qwen results use 10 seeds, whereas Falcon results generally use 3 seeds unless otherwise noted.
  • The primary Qwen comparison uses S0 at 12.6M parameters versus rank-24 LoRA at 4.7M, while matched-budget rank-64 LoRA degrades by −15.5 pp.The Falcon comparison is described as supportive rather than definitive because it uses 3 seeds.
  • The study treats Qwen3.5 HumanEval as the anchor claim, with Falcon testing architectural transfer and MATH-500, GSM8K, and Spider testing domain transfer.

5 Results

S0 improves HumanEval performance across hybrid architectures, scales strongly with model size, transfers to math benchmarks, and requires architecture-specific state scaling. Its gains are not explained by parameter count alone, while transfer fails on Spider and prefix tuning fails on a pure Transformer.

  • Main comparison: +23.6 ± 1.7 pp: S0 raises Qwen3.5-4B greedy HumanEval pass@1 from 48.8% to 72.2% over 10 seeds.Against rank-24 LoRA, S0 gains +10.8 pp with roughly three times lower variance.
  • Cross-architecture comparison: 71.8%±1.3: S0 matches LoRA’s 71.4%±2.4 on FalconH1-7B, with results statistically indistinguishable across 3 seeds.The comparison is supportive evidence rather than a definitive superiority claim.
  • Parameter-matched control: +39 pp: at a matched 12.6M-parameter budget, S0 gains +23.6 pp while LoRA rank 64 degrades by −15.5 ± 18.9 pp.Eight of ten rank-64 LoRA seeds were negative, consistent with overfitting in this small-data regime.
  • Scaling behavior: +44.0 pp: Qwen3.5 gains increase from +2.6 pp at 0.8B to +44.0 pp at 9B, although non-monotonic baselines complicate scaling analysis.The 9B model reaches 76.1% from a 32.1% baseline.
  • Cross-domain transfer: +4.8 ± 1.4 pp on MATH-500 and +2.8 ± 1.6 pp on GSM8K are significant cross-domain gains, while Spider shows +0.0 pp transfer.Spider is a single-seed alpha sweep and is treated as a boundary-condition observation.
  • Controls and trade-offs: −13.9 pp: matched-parameter prefix tuning on pure-Transformer Qwen2.5-3B degrades performance under all nine tested configurations.The per-step state-offset variant reaches +27.1 pp on Qwen3.5 but incurs per-step inference cost.

6 Mechanistic Analysis

Mechanistic analyses indicate that S0’s direct prompt influence decays through recurrence but remains sufficient to alter generation immediately. The resulting first-token divergence is amplified autoregressively into different solution trajectories, with gating differences requiring architecture-specific scaling.

  • Persistence analysis: 0.03% KL ratio: S0’s direct influence on output logits decays to this level by the final prompt token.The measured decay matches the predicted exponential profile from the recurrent gating analysis.
  • First-character divergence: 23 of 27 FAIL-to-PASS flips (85%) diverge at the first generated character, and all 27 diverge within the first 10% of completion.A sign test for concentration within the first 10% yields p < 10^-8.
  • Trajectory steering: S0 shifts the output distribution at the first opportunity, after which autoregressive decoding amplifies the initial divergence into a qualitatively different solution.This trajectory-steering mechanism differs from LoRA’s uniform weight modification.
  • Architecture-specific gating: 10× alpha gap: Qwen3.5 uses 0.07 while FalconH1 uses 0.65, showing that initial-state scale does not transfer unchanged across recurrence families.The architectures have different gating statistics and mechanisms.
  • Representation analysis: A correctness probe reaches AUC = 0.93 from the recurrent state versus AUC = 0.90 from the residual stream.The probe uses PCA-reduced intermediate representations.

7 Related Work

Prior PEFT work largely favors LoRA on diagonal-state architectures, whereas this paper studies matrix-valued recurrent states in hybrid models. It reports stronger initial-state tuning results across GatedDeltaNet and Mamba-2 and distinguishes S0 from related state-based approaches.

  • Prior PEFT findings: On GatedDeltaNet and Mamba-2, S0 outperforms LoRA by +10.8 pp on Qwen3.5, reversing the diagonal-state pattern reported in prior SSM PEFT work.The authors relate this reversal to the greater expressiveness of full matrix states.
  • State structure: Full matrix states can encode cross-feature correlations, unlike Mamba-1’s diagonal state, motivating a state-expressiveness explanation for differing PEFT outcomes.A single Qwen3.5 GatedDeltaNet layer carries approximately 524K state entries.
  • Conceptual distinction: State Soup interpolates in-context-learned Mamba-2 states, whereas S0 directly optimizes initial states by gradient descent.The approaches differ in whether states arise from interpolation or direct optimization.
  • Related state-based methods: The per-step state-offset variant reaches +27.1 pp on GatedDeltaNet, while pure S0 trades peak accuracy for zero inference overhead.The paper extends the offset idea to GatedDeltaNet and reports no prior validation of state-based PEFT across multiple recurrence families in one study.

8 Discussion and Limitations

S0 transfers strongly from HumanEval to math benchmarks but not Spider, while its scope depends on expressive matrix-valued states and verified supervision. The supporting experiments are deliberately narrower than a complete benchmark sweep.

  • Transfer: +4.8 pp on MATH-500 and +2.8 pp on GSM8K are statistically significant, whereas Spider text-to-SQL shows +0.0 pp transfer.The paper reports p=0.00002 for MATH-500 and p=0.0003 for GSM8K; Spider is identified as a structured-output failure case.
  • Scope: Initial-state tuning underperforms LoRA on diagonal-state Mamba-1, whereas this study evaluates matrix-valued GatedDeltaNet and Mamba-2 states.The authors connect this contrast to a state-expressiveness threshold below which initial-state tuning cannot compete with weight adaptation.
  • Training data: S0 requires execution-verified correct solutions, with roughly 48 verified HumanEval completions used in the main setup.Applying S0 where verification is expensive remains untested, and LoRA does not impose this requirement.
  • Evidence strength: Falcon uses 3 seeds, Spider uses a single-seed alpha sweep, and the study presents these as transfer checks rather than a complete benchmark sweep.The paper attributes the limited grid to a fixed personal GPU budget and frames the thinner comparisons as boundary conditions.

9 Conclusion

S0 tunes only recurrent initial states in hybrid models and adds no inference overhead. It is competitive with LoRA across the reported Qwen and Falcon comparisons, while layer sensitivity identifies early layers as especially effective.

  • Conclusion: S0 tunes only the initial recurrent state of hybrid language models, adding zero inference overhead.The conclusion presents this as the method’s central operating property.
  • Conclusion: S0 outperforms LoRA by +10.8 pp on Qwen3.5-4B and is statistically indistinguishable from LoRA on FalconH1-7B at 71.8% versus 71.4%.The Falcon comparison uses 3 seeds, while the Qwen result is the central comparison.
  • Conclusion: A prefix-tuning control on a pure Transformer fails under all configurations tested, while S0 transfers significantly to MATH-500 and GSM8K but not structured-output SQL.These comparisons bound the reported scope to hybrid recurrent-attention models and selected transfer tasks.
  • Layer sensitivity: Early layers 0–7 alone recover +23.8 pp, matching the full-model +22.6 pp with 4.2M rather than 12.6M parameters.Late layers yield the smallest gain, +13.1 pp, but produce zero degradations.

D Training Data Efficiency

S0 remains data-efficient with stable gains from 25 verified solutions, while alpha scaling is architecture-specific and higher-rank LoRA remains unstable or harmful in the matched experiments.

  • Training data efficiency: +22.6 ± 3.2 pp is achieved with 25 correct solutions, and 50 solutions provide the same mean with no further benefit.Ten solutions already yield +25.4 ± 6.8 pp, but with higher variance.
  • Alpha scaling: +30–+34 pp sampled gains occur for α=0.03 through α=0.15 on Qwen3.5, peaking at +33.9 pp for α=0.07.These sampled results exceed the greedy 10-seed mean because temperature-0.7 sampling amplifies the learned prior’s effect.
  • Alpha scaling: Falcon reaches 71.8% accuracy at α=0.60–0.70, whereas α=0.07 yields only +8.3 pp, showing architecture-specific state scaling.The experiments do not isolate which update component drives the alpha shift.
  • Parameter-matched LoRA: Rank-64 LoRA, matched to S0’s 12.6M parameters, has 8 of 10 negative seeds, with deltas from −41.7 pp to +13.1 pp.The experiments use identical training data, learning rate, and steps across the higher-rank comparisons.
  • Pass@k: S0 exhibits low pass@k variance across seeds, while LoRA pass@5 and pass@10 remain within baseline noise.These aggregate and per-seed results are reported for three seeds each.

H Probing Protocol and Scaling Data

Recurrent states encode task-relevant information at least as strongly as residual-stream activations in the probe, while larger Qwen3.5 models show clearer gains than the smallest model tested.

  • Probing protocol: AUC=0.93 from recurrent states exceeds AUC=0.90 from residual-stream hidden states in the logistic-regression probe.The best recurrent-state layer is 18, compared with layer 23 for the residual stream.
  • Probing protocol: The 2.5-point AUC gap is directional evidence that recurrent states encode task-relevant information at least as strongly as residual-stream activations.The probe uses final-prompt-token representations projected to 64 dimensions via PCA and 5-fold cross-validation.
  • Scaling data: The 0.8B Qwen3.5 result is borderline at p=0.076, while gains at 4B and above are large and unambiguous.The scaling comparison uses HumanEval results across Qwen3.5 model sizes.
Loading 2604.01168v2…