PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

Nux Li

arXiv:2609.30820v1cs.LGcs.CL

TL;DR

Looped models reuse weights across recurrence, but conventional single-pass quantization and calibration may not reflect how perturbations and inputs evolve across steps. The paper studies where quantization error enters the recurrent state and calibrates GPTQ along recurrence trajectories; trajectory Hessians recover performance in every tested case, while one-step GPTQ trails RTN on five of nine checkpoints.

  • Problem

    Conventional calibration-based post-training quantization uses single-forward-pass activations, leaving open how well it serves looped models that reuse weights across recurrence steps.

    Source: §1 · p. 1, §8 · p. 10

  • Method

    The paper uses controlled interventions to study recurrent-transition perturbations and compares GPTQ calibrated on first-invocation activations with GPTQ using Hessians accumulated across recurrence steps.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    One-step GPTQ underperforms RTN on five of nine checkpoints, while accumulating Hessians along the recurrence trajectory recovers performance in every tested case.

    Source: §1 · p. 1, §8 · p. 10

  • Takeaways & Limitations

    For looped models, quantization must account both for where perturbations enter feedback and for which states calibration observes.

    Source: §8 · p. 10

    Evaluation covers nine checkpoints up to 4.17B parameters, and trajectory calibration does not always recover full-precision performance.

    Source: §8 · p. 10

Abstract

from arXiv · show

Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes of standard post-training quantization. On Huginn-3.5B, per-channel INT4 fails primarily at the non-residual loop-entry adapter, while quantizing the residual core is much less damaging. We call this feedback exposure: a quantized layer perturbs the recurrent state without an identity path, and the resulting error is fed back at later steps. Controlled experiments on linear filters and Mamba state-space models show that feedback exposure also occurs outside transformers. Grouped INT4 reveals a separate failure, calibration blindness: our one-step GPTQ baseline builds its Hessian from step-0 activations, leaving input directions used later in the recurrence nearly unweighted. Across nine checkpoints from seven looped architectures, one-step GPTQ is worse than round-to-nearest (RTN) on the primary task metric for five checkpoints. Accumulating the GPTQ Hessian across recurrence steps outperforms both one-step GPTQ and RTN on all nine checkpoints and recovers bf16-level accuracy on Huginn. These results separate two questions for PTQ on looped models: where quantization error enters the recurrence, and which states calibration sees.

1 Introduction

The paper distinguishes two quantization failures in looped models: perturbations entering an unprotected recurrent transition and calibration that misses later-used input directions. Huginn diagnostics localize severe INT4 damage to the loop-entry adapter, while trajectory calibration addresses the separate one-step GPTQ mismatch.

1 Introduction

  • Looped models reuse shared weights across recurrence steps, motivating tests of whether single-pass INT4’s roughly quarter-sized weight footprint transfers to them.Huginn-3.5B unfolds to effective depth 132 at 32 recurrences; Ouro models reuse stacks for four loop steps, while COCONUT recycles GPT-2’s forward pass.Source: §1 · p. 1
  • Adapter-only INT4 ends at 10.5% token-prediction agreement with bf16, versus 87.9% for core-only INT4 on the 50-prompt diagnostic.Quantizing both components yields 10.6% agreement; per-channel INT4 on both reduces GSM8K exact match from 34% bf16 to 0% on a separate 100-example diagnostic.Source: §1 · p. 1
  • Standard sensitivity metrics miss where rounding error enters the recurrence, failing to explain why the adapter is more damaging than the recurrent core.Small random perturbations rank the adapter least locally sensitive, while Hessian trace ranks a core projection most sensitive.Source: §1 · p. 1
  • One-step GPTQ underperforms RTN on five of nine checkpoints despite grouped INT4 avoiding the immediate collapse.Its step-0 Hessian misses later-used directions, and the mismatch persists with more calibration prompts.Source: §1 · p. 1
  • The paper treats feedback exposure and calibration blindness as distinct issues, using controlled interventions for the former and trajectory calibration across nine checkpoints from seven architectures for the latter.Huginn’s adapter lies at their intersection.Source: §1 · pp. 1–2

2 Background

This background defines the recurrent state and architecture variations relevant to quantization, then frames calibration and stability diagnostics. It positions trajectory-aware calibration against prior work on iterative systems and explains why feedforward residual robustness does not directly characterize repeated perturbations.

2 Background

  • A looped transformer repeatedly applies a shared block to latent state s_t, with fixed prompt embedding e and architecture-specific conditioning c_t.Huginn and R-Llama use a non-residual loop-entry adapter, while other models reuse residual layers or impose explicit stability constraints.Source: §2 · p. 2
  • The local spectral radius ρ of the per-step state Jacobian is a local stability surrogate; ρ > 1 indicates linearized amplification in some direction.The echo state property requires trajectories driven by the same input to forget their initial conditions; uniform contractivity is sufficient globally.Source: §2 · p. 2
  • GPTQ rounds weights column by column to minimize calibration-output error, using H = X⊤X as a per-layer Hessian proxy; the paper compares first-invocation and all-invocation Hessians.At group size 128, each group of consecutive columns receives its own scale; orthogonal transforms and activation-based rescaling are alternative PTQ approaches.Source: §2 · p. 2
  • Prior calibration methods sample or weight recursive denoising timesteps, while this paper addresses shifts from feeding a shared block’s output back as its next latent state.For linear systems, cited work bounds quantized error when the transition matrix is unperturbed; the studied checkpoints lack the cited convergence certificate for monotone DEQs.Source: §2 · p. 2
  • Feedforward residual path-counting does not describe looped models because the same weight perturbation acts on the recurrent state at every step.In the cited feedforward argument, each residual block appears at most once on any path.Source: §2 · p. 2

3 Feedback Exposure Under Recurrence

Per-channel INT4 is far more damaging at Huginn’s non-residual loop-entry adapter than in its residual core. Controlled interventions and cross-system tests link this asymmetry to perturbations that enter the recurrent state without an identity path and are fed back across steps.

3.1 Controlled Injection

  • Adapter-only INT4 leaves 10.5% final-step agreement with bf16, versus 87.9% when all 16 recurrent-core projections are quantized.Quantizing both components yields 10.6%, matching adapter-only INT4.Source: §3.1 · p. 3, Figure 1 · p. 3
  • Removing prior-state error raises final-step agreement from 10.4% to 69.2%, while relative final-state error grows monotonically with the injected error fraction λ.The trend holds for both per-channel and grouped INT4.Source: §3.1 · p. 3
  • R-Llama shows the same adapter/core asymmetry, while COCONUT’s agreement rises from 44.8% after its base forward to 61.5% after eight latent recurrences.R-Llama adapter-only agreement is 49.1%, versus 74.8% for core-only; COCONUT’s residual stack has no dedicated adapter.Source: §3.1 · pp. 3–4

3.2 Feedback Exposure and the Identity Path

  • The adapter’s linear map perturbs the recurrent state without an identity bypass, whereas each residual core block preserves an additive identity path.The paper calls a perturbation feedback-exposed when it acts on the recurrent state without an identity path around it.Source: §3.2 · p. 4
  • Feedback exposure also appears beyond transformers: quantizing transition matrix A moves a pole outside the unit circle in all four tested filters, and INT4 perturbations to Mamba’s A_log increase state error.Quantizing B, C, or D leaves the poles unchanged across six tested precisions; perturbing Mamba’s out_proj leaves that layer’s SSM state unchanged.Source: §3.2 · p. 4
  • Keeping Huginn’s adapter in bf16 recovers 7 of the 9 exact-match points lost to grouped-INT4 quantization in a 100-example GSM8K ablation.At per-channel INT4, adapter-only Hadamard rotation with trajectory calibration reaches 29.4% exact match on the full 1,319-example GSM8K test set, versus 34.9% in bf16.Source: §3.2 · p. 5

4 Calibration Blindness from Step-0 Hessians

One-step GPTQ builds its Hessian from step-0 activations, which can leave later-used input directions weakly weighted. The paper measures this calibration mismatch across looped models and finds that rank coverage can shift substantially across recurrence steps and layers.

4 Calibration Blindness from Step-0 Hessians

  • One-step GPTQ underperforms round-to-nearest on five of nine checkpoints because its step-0 Hessian can leave later-used input directions weakly penalized.The baseline minimizes a Hessian-weighted proxy built from step-0 activations alone.Source: §4 · p. 5
  • The Huginn adapter’s thresholded Hessian rank rises from 43 at step 0 to 335 after accumulating 32 recurrence steps, out of 10,560 input dimensions.The table reports per-layer thresholded rank at 10^-3λmax on 50 GSM8K prompts.Source: Table 1 · p. 5, §4 · p. 5
  • With the same 2,919 prompt-token positions, Huginn’s rank rises from 43 at step 0 to 259 at step 1 and 363 at step 4.Ouro and LoopFormer also show large first-step jumps, while COCONUT changes little.Source: §4 · p. 5
  • Low step-0 rank alone does not measure trajectory shift: the adapter gains 7.8× rank coverage, while block-3 mlp.proj gains 1.4× despite similarly low initial rank.For Huginn’s adapter, only 43 of 10,560 input dimensions exceed the step-0 threshold.Source: §4 · p. 5
  • Ouro-1.4B’s layer-0 down_proj has step-0 rank 1/5,632, yet four-step calibration improves PPL from 14.67 to 12.97.Later-step Hessians individually reach rank 974–1,194, though their thresholded accumulated rank remains 1.Source: §4 · pp. 5–6

4.1 Cross-Architecture Rank Survey

  • All nine checkpoints have at least 2× maximum rank gain, and rank gain measures accumulated-to-step-0 rank coverage for a shared layer.Huginn, Ouro, and LoopFormer have later-step input shifts associated with large gains; COCONUT’s per-step ranks barely shift (43 to 50).Source: §4.1 · p. 6, Figure 2 · p. 6

5 Trajectory Calibration

Trajectory calibration builds a recurrence-wide reconstruction proxy rather than relying on step-0 activations alone. The results show that full-rollout calibration is not always necessary, and that rank gain alone does not determine recovery.

5 Trajectory Calibration

  • Accumulating Hessians across the rollout adds covariance from later-used directions, so GPTQ penalizes rounding error along directions used during deployment.The reconstruction proxy holds the state trajectory fixed at the bf16 reference.Source: §5 · p. 7
  • Under grouped INT3, recollecting Huginn Hessians on the quantized rollout changes held-out final-step agreement from 84.59% to 84.95%, with a 95% CI of [−2.31, 3.10] pp.The grouped-INT4 reconstruction proxy remains nearly constant in magnitude along the bf16 rollout even as active directions change.Source: §5 · p. 7
  • COCONUT’s thresholded Hessian rank changes little across steps, with a 1.1× maximum gain, and trajectory accumulation improves in-sample top-1 agreement by only 0.8 percentage points.Source: §5 · p. 7
  • On Huginn WikiText-2, step-31-only calibration reaches 15.07 PPL, compared with 15.36 for one-step and 14.89 for full accumulation.Step 31 captures 62% of the one-step-to-full improvement, but its higher thresholded rank does not by itself determine recovery.Source: §5 · p. 7
  • At the full 32-step horizon, Huginn GSM8K exact match is 30% versus 25% at Ncal=1 and 8, 28% for RTN, and 32% for bf16.The horizon curves differ across architectures, so the main table uses each model’s deployed recurrence depth.Source: §5 · p. 7

6 Evaluation

The evaluation tests trajectory calibration across nine checkpoints from seven looped architectures, using grouped INT4 and task-appropriate metrics. Trajectory calibration improves every reported checkpoint, while a real-kernel evaluation measures Huginn’s latency, storage, and accuracy.

6 Evaluation

  • The evaluation covers nine checkpoints from Huginn, R-Llama, Ouro, LoopFormer, Parcae, PonderPythia, and Nanbeige; recovery uses deterministic simulated grouped INT4 (g=128).Huginn and R-Llama use GSM8K accuracy, while the other checkpoints use WikiText-2 perplexity.Source: §6 · p. 7

6.1 Performance Recovery

  • Trajectory calibration improves every row: Huginn returns to bf16-level accuracy, while the other checkpoints close 17–96% of the RTN-to-base gap.One-step GPTQ underperforms RTN on five checkpoints; on LoopFormer, the deficit is 10.81 PPL.Source: §6.1 · p. 7, Table 2 · p. 8
  • At 1.3B parameters, Parcae PPL is 20.84 with trajectory calibration, 27.35 with one-step GPTQ, 51.30 with RTN, and 19.57 with f32.Across other checkpoints, the supplied results include 79.8% GSM8K for Ouro-2.6B with 4-step calibration versus 78.8% for RTN and 72.6% for one-step; PonderPythia-2.8B reaches 12.24 PPL versus 12.51 for one-step and 12.61 for RTN.Source: §6.1 · pp. 8–9
  • On full-test GSM8K at grouped INT3, exact match is 20.70% with full-trajectory GPTQ, 17.44% with equal-row trajectory calibration, 8.11% with RTN, and 5.84% with one-step GPTQ.Source: §6.1 · p. 9

6.2 Real-Kernel Evaluation

  • Marlin W4A16 reduces 32-step Huginn latency from 303.9 ms in bf16 to 250.9 ms, a 1.21× speedup, and reduces shared-projection weight storage from 3.27 GB to 0.86 GB.On full GSM8K, accuracy is 34.0% under Marlin and 35.0% under deterministic simulated INT4 with the same quantized weights.Source: §6.2 · p. 9

7 Related Work

The related work spans quantization, iterative calibration, recurrent systems, and residual-network stability. The paper distinguishes its analysis by separating feedback exposure from calibration blindness and testing the mechanisms directly.

7 Related Work

  • Prior looped-transformer work includes accumulated GPTQ Hessians, loop-dependent quantization methods, and reports of architecture-dependent INT4 fragility; the paper separates mechanisms through causal analysis.Rank and subspace measurements identify directions missed by step-0 Hessians, while controlled changes isolate error recirculation and residual protection.Source: §7 · p. 9
  • Q-Diffusion and AdaTSQ address activation shifts along denoising trajectories, while looped transformers reuse the same operator to generate and process later states.Source: §7 · p. 9
  • Quamba’s stated bound assumes an unperturbed transition matrix, whereas the paper’s IIR and Mamba controls perturb the transition itself.The IIR control keeps the pole spectrum fixed when quantizing B, C, and D.Source: §7 · p. 9
  • Residual-network stability analyses do not cover repeated re-entry of the same perturbation; the paper extends the stability view to discrete recurrence and tests identity-bypass protection.Source: §7 · p. 9
  • Single-pass PTQ methods typically calibrate without recurrence-conditioned activations; GPTQ and LDLQ inherit step-0 Hessian rank deficiency, which orthogonal QuIP# rotation does not restore.AWQ selects per-channel scales from activation magnitudes, which are more stable across recurrence steps.Source: §7 · p. 9

8 Conclusion

The conclusion identifies feedback exposure and calibration blindness as distinct issues in recurrent weight reuse. It reports recovery across the tested checkpoints but notes that full-precision performance is not always recovered.

8 Conclusion

  • In looped transformers, rounding error is dangerous when it enters a recurrent transition without an identity path; IIR and Mamba controls likewise show greater damage when quantization changes the transition.Source: §8 · p. 10
  • One-step GPTQ underperforms RTN on five of nine checkpoints, while accumulating Hessians along the recurrence recovers performance in every tested case.Source: §8 · p. 10
  • The evaluation covers nine checkpoints up to 4.17B parameters, and trajectory calibration does not always recover full-precision performance.Rank gain alone does not predict recovery magnitude.Source: §8 · p. 10

A Architecture Diagram

  • The Huginn adapter projects the concatenated recurrent state and input back to model width without a residual path, while core residual paths provide an identity channel.Source: Figure 4 · p. 13

B Trajectory Calibration Algorithm

C Supporting Tables

  • At Parcae Ncal=2, block-0 mlp.proj has 3 of 6,144 eigenvalues above 10^-3λmax and GPTQ gives worse PPL than RTN (99.59 versus 51.30); at Ncal=4, PPL recovers to 20.98.PPL rises slightly from 20.62 at Ncal=6 to 20.84 at Ncal=8.Source: C Supporting Tables · pp. 13, 15

D Rotation and Marlin Implementation Checks

Rotation variants are near-identical at INT8, while deployment checks find close agreement between Marlin and FakeQuant and small projection-level reconstruction error.

D Rotation and Marlin Implementation Checks

  • The rotation comparisons use adapter-only rotation, and the per-channel diagnostic is reported in Table 14.Simulated evaluations are deterministic; the Marlin deployment diagnostic uses the real kernel.Source: D Rotation and Marlin Implementation… · p. 16, Table 14 · p. 17
  • All tested INT8 rotation variants are within 0.01 PPL of bf16: Hadamard and random orthogonal reach 14.02, while block-size-32 Hadamard reaches 14.03.The QR fallback for partial Hadamard blocks removes the remaining INT8 perplexity difference.Source: D Rotation and Marlin Implementation… · p. 16
  • Marlin reaches 34.0%/34.3% strict/flexible accuracy versus FakeQuant’s 35.0%/35.3%, with paired differences of −1.0/−1.1 percentage points (McNemar p = 0.28/0.24).Both use identical quantized weights and scales across all 17 shared projections; the diagnostic calibrates on 50 GSM8K training questions and evaluates 1,319 test questions.Source: D Rotation and Marlin Implementation… · p. 17
  • Across 51 shared-projection comparisons at 1, 7, and 16 input tokens, maximum relative ℓ2 error is 1.35 × 10−3.A 32-step forward at each input length executes the expected 544 Marlin calls.Source: D Rotation and Marlin Implementation… · p. 17

E Extended Controlled Injection Analysis

Controlled interventions show that adapter error grows when prior state error is fed back, while non-residual-path perturbations are more damaging across tested perturbation types. Grouped INT4 retains high agreement compared with per-channel quantization.

E Extended Controlled Injection Analysis

  • Removing prior state error raises final-step agreement from 10.4% to 69.2%, while relative final-state error grows monotonically with feedback strength under per-channel and grouped INT4.For the full adapter, the λ = 1-to-λ = 0 final-state error ratio is 1.989× under per-channel INT4 and 2.085× at g=128.Source: E Extended Controlled Injection Analysis · p. 17
  • Adapter-only INT4 gives 10.5% top-1 agreement versus 10.6% when both adapter and core are quantized, so core INT4 adds little damage beyond adapter error.The controlled injection result is consistent with the adapter term remaining dominant whether or not core blocks are quantized.Source: E Extended Controlled Injection Analysis · p. 17

F Perturbation-Type Generalization

  • Rank-4 perturbations preserve a 24.0-point adapter/core agreement gap, and trained updates widen it to 64.4 points; Gaussian perturbations show the same ordering despite smaller adapter norm.Core-block perturbations are less damaging in all three settings, consistent with residual-path buffering; Table 15 gives the evaluated prompt and seed scope.Source: F Perturbation-Type Generalization · pp. 17–18, Table 15 · p. 18

G Sensitivity Metric Comparison

  • The adapter is a 2.3× ablation-sensitivity outlier but ranks last by random-direction gain and third by Hessian trace.Ablation sensitivity measures changed top-1 predictions after quantizing one projection; random-direction gain measures output change per input perturbation, and tr(H) is GPTQ’s activation Hessian trace.Source: Table 16 · p. 18, G Sensitivity Metric Comparison · p. 18

H Group-Size Dependence

I Cross-Architecture Rank Survey

The cross-architecture rank survey finds that step-0 calibration can miss bottleneck directions used later in recurrence. Proxy, Jacobian, R-Llama, and IIR analyses characterize how this mismatch and recurrent perturbations behave across settings.

I Cross-Architecture Rank Survey

  • Ouro bottlenecks remain at thresholded rank 1 after adding recurrence steps, even though later-step Hessians individually reach ranks of 974–1,194 and 1,010–1,335.The maximum rank gain can belong to a non-bottleneck layer; Parcae’s bottleneck drives per-layer calibration failure despite one-step GPTQ beating RTN at model level, while Ouro-2.6B’s model average would not flag its bottleneck.Source: I Cross-Architecture Rank Survey · p. 19, Table 18 · p. 19
  • In a separate Huginn control holding the same 2,919 prompt-token positions across steps, ranks were 361–363 at steps 7, 15, and 31, each slightly above the 32-step accumulated rank of 335.Source: I Cross-Architecture Rank Survey · p. 19

J Frozen-Trajectory Proxy Stability

  • At g=128, the frozen-trajectory proxy varies only 1.01× across Huginn’s 32 steps, and step 0 contributes 3.2% of the cumulative proxy, near the 3.125% uniform expectation.Only 43 of 10,560 adapter input directions exceed the step-0 threshold (0.41%), indicating a directional calibration gap despite roughly equal proxy magnitude across steps; the per-channel adapter proxy is used diagnostically because ρ > 1 on all 30 prompts.Source: J Frozen-Trajectory Proxy Stability · pp. 19–20
  • For Ouro-1.4B, the g=128 per-step proxy varies 1.02× across four steps, with step 0 contributing 25.2% against a 25.0% uniform expectation.The proxy is computed on bf16 reference activations with the state trajectory fixed.Source: J Frozen-Trajectory Proxy Stability · pp. 19–20

K Jacobian Measurement Details

  • At the bypass bf16 state, the INT4-induced spectral-radius shifts are −0.340 without and −0.041 with bypass, a difference-in-differences of +0.299.At the no-bypass bf16 state, the corresponding difference-in-differences is +0.021 with a 95% CI of [−0.061, 0.121], so the effect depends on operating point.Source: K Jacobian Measurement Details · p. 21
  • At the no-bypass bf16 step-32 state, adding the identity bypass attenuates the bf16/INT4 response mismatch by 41.66%.The bypass comparison uses matched states, weights, and directions; across tested radii, the direction holds, but the untrained bypass changes the bf16 rollout, so the experiment supports only the local structural claim.Source: K Jacobian Measurement Details · p. 21

L Error Amplification: R-Llama

  • R-Llama’s final mean displacement is 1082 under per-channel INT4 versus 300 at g=128, a 3.6× reduction matching the weight-error norm ratio.Both displacement curves are within 1% of their final value after four updates; per-channel RTN gives 0% GSM8K accuracy, while g=128 RTN retains 39.5%.Source: L Error Amplification: R-Llama · p. 21

M IIR Filter Experimental Details

  • Quantizing IIR transition matrix A pushes at least one pole outside the unit circle for all four tested designs at 10 bits or below, while B, C, and D quantization leaves poles unchanged.The order-6 Chebyshev boundary case remains unstable even at 16 bits; the Butterworth design stabilizes at 12 bits.Source: M IIR Filter Experimental Details · p. 22, Table 20 · p. 22

N Mamba SSM Experimental Details

Mamba experiments compare INT4 and norm-matched Gaussian perturbations across layers, measuring hidden-state divergence through the scan. Transition perturbations produce growing divergence, and identity injection reduces A_log sensitivity.

N Mamba SSM Experimental Details

  • The experiments test FakeQuant with each component’s actual INT4 error norm against Gaussian noise matched to the median INT4 error across components.Hidden-state L2 divergence is measured over ten approximately 100-token prompts at five layers per model size; Gaussian runs use three seeds per layer.Source: N Mamba SSM Experimental Details · pp. 22–23
  • Under INT4 FakeQuant, A_log perturbation divergence grows 1.28–2.00× at 130M and 1.30–1.71× at 2.8B, while out_proj causes zero measured SSM-state divergence.A_log divergence starts at zero because the perturbed transition term has no state contribution at t=0; in_proj and x_proj produce divergence immediately.Source: N Mamba SSM Experimental Details · p. 23
  • Adding an identity term reduces A_log sensitivity monotonically from α = 0.15 onward, reaching a 20% divergence reduction at α = 0.5.The sweep uses Mamba-130M layer 12; gains diminish beyond approximately α = 0.25.Source: N Mamba SSM Experimental Details · p. 23, Table 21 · p. 23

O Eigenvalue Spectrum Analysis

The adapter’s step-0 calibration spectrum is narrow compared with the core, while accumulating recurrent-step activations broadens adapter coverage. Excluding step 0 helps some checkpoints but hurts others, and per-layer proxy gains do not necessarily improve model-level perplexity.

O Eigenvalue Spectrum Analysis

  • 32-step accumulation raises adapter rank 7.8× (43 → 335), while core step-0 rank rises from 2,111 to 3,588 as prompts increase from 50 to 1,000.The adapter’s maximum eigenvalue changes by less than 0.3% across prompt counts, so its rank decrease is not caused by a growing threshold.Source: Figure 5 · p. 24, O Eigenvalue Spectrum Analysis · p. 25
  • Step-0 and step-31 adapter subspaces have only 0.571% and 0.631% mean squared-cosine overlap across two seeds, while step-31 subspaces overlap 99.71% across seeds.The step-0 subspace is neither aligned with the later-step subspace nor stable across initializations.Source: O Eigenvalue Spectrum Analysis · p. 25

P Step-Window Ablation

  • Skip-0 changes WikiText-2 PPL by −0.52 on Ouro-2.6B and −0.26 on Parcae-1.3B, but worsens LoopFormer (+0.25) and Ouro-1.4B (+0.81).Huginn (+0.03) and R-Llama (+0.19) do not benefit.Source: Table 23 · p. 25, P Step-Window Ablation · p. 25
  • For Ouro-2.6B, PPL worsens monotonically from 11.10 to 11.60 as step-0 weight α rises from 0 to 0.25; Ouro-1.4B instead has an optimum at α=0.15.Skip-0 is a per-checkpoint tuning option; full accumulation remains the default.Source: P Step-Window Ablation · p. 25
  • Skip-0 lowers per-layer tr(∆W H3 ∆W⊤) in 335/336 Ouro-2.6B layers and 168/168 Ouro-1.4B layers, but PPL improves only at 2.6B.The frozen-trajectory proxy fixes activations at the bf16 reference and drops cross-step terms, so per-layer gains need not translate to lower model-level perplexity.Source: P Step-Window Ablation · pp. 25–26

Q Experimental Setup

  • GSM8K is the primary generation benchmark for Huginn and R-Llama, while WikiText-2 perplexity is primary for the listed language-model checkpoints; main-table quantization uses grouped INT4 with g=128.GSM8K uses 8-shot chain-of-thought evaluation with 1,319 examples; main-table FakeQuant isolates calibration effects from integer-kernel behavior.Source: Q Experimental Setup · p. 26

Related papers

Loading 2609.30820v1…