PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma

arXiv:2610.06833v1cs.LG

TL;DR

Each recurrence adds cost to looped language models, while whether they should learn fixed points remains contested. The paper changes the depth prior and input injection to shape fixed points. Across model scales, these changes improve prediction quality, and at 1.6B the learned prior matches fixed-depth training’s downstream average with a 3× smaller KV cache.

  • Problem

    Each recurrence adds cost in training, decoding, prefill, and RL, while whether looped language models should learn fixed points remains contested.

    Source: §1 · p. 2

  • Method

    The paper learns a depth prior from prediction feedback with an entropy term and uses orthogonal input injection to remove the state component along the input.

    Source: §1 · p. 2

  • Results

    From 100M to 1.6B, the learned prior lowers perplexity and raises downstream average over Huginn’s fixed prior; at 1.6B, it matches fixed-depth training’s full-cache downstream average with a 3× smaller KV cache.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Near fixed points, endpoints can replace trajectories across training, inference, and post-training, cutting memory or time at a small quality cost.

    Source: §7 · p. 14

    Experiments use one seed per configuration and models up to 1.6B parameters; larger scales, other configurations, and real-world feasibility remain untested.

    Source: §7 · p. 14

Abstract

from arXiv · show

Every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning (RL). The closer recurrent states get to fixed points, the less the path to them matters. This enables truncated backpropagation in training; terminal key-value (KV) sharing for decoding with almost no loss in accuracy; a distilled student that prefills up to 1.79x faster; and RL updates that compute gradients from saved rollout states, 2x faster than backpropagating through the replayed trajectory. We therefore improve the two components of training that shape these fixed points: the depth prior and input injection. Fixed-depth training breaks KV sharing, and Huginn's broad depth prior supports sharing but dilutes supervision at the target depth more than sharing requires; we learn the prior from prediction feedback, with an entropy term that keeps it broad. Existing injection schemes let the state's component along the input amplify or cancel the injection; we remove this component with orthogonal injection. From 100M to 1.6B parameters, the learned prior and orthogonal injection lower perplexity at every scale relative to Huginn's prior and existing injection schemes, respectively. At 1.6B, the learned prior with a 3x smaller KV cache matches the downstream average of fixed-depth training with the full cache.

1 Introduction

Fixed points let a looped language model use its endpoint instead of paying for every step of the recurrent trajectory. The paper develops fixed-point shortcuts and changes training to shape the fixed points those shortcuts require.

1 Introduction

  • Fixed points make endpoint-based shortcuts possible across training, decoding, prefill, and RL, where each recurrence otherwise adds cost.States approach fixed points in Huginn but not Ouro, matching the contrast in how terminal KV sharing affects their zero-shot GSM8K accuracy.Source: §1 · p. 2
  • At 1.6B, terminal KV sharing drops fixed-depth full-BPTT GSM8K accuracy from 50.6 to 21.2.With tolerant fixed points, a 3× smaller cache keeps entropy-regularized learned-prior models within 0.4 points of their full-cache downstream average.Source: §1 · p. 2
  • Distilled prefill is up to 1.79× faster on 8K prompts, while rollout-state reuse halves RL scoring and backward time per update.The distilled student uses a quarter of the teacher’s pretraining tokens and trails its downstream average by 1.0–4.5 points; rollout-state reuse stays within 1.6 points of the replay baseline’s GSM8K pass@1.Source: §1 · p. 2
  • From 100M to 1.6B, the learned depth prior lowers perplexity and raises downstream average over Huginn’s fixed prior.At 1.6B, it matches the full-cache downstream average of fixed-depth training with a 3× smaller KV cache.Source: §1 · p. 3
  • OrthoInj lowers training loss, reduces validation perplexity by 0.3–1.4%, and raises downstream average over Parcae from 100M to 1.6B.It removes the state component along the input so each recurrence receives the input at the same strength.Source: §1 · pp. 2–3

2 Fixed Points in Looped Language Models

The section defines a looped model as repeated application of a shared update map to a recurrent state. It measures convergence token by token and finds that tokens reach approximate fixed points at different depths.

2 Fixed Points in Looped Language Models

  • The recurrent core iterates a shared update map from an initial state; a fixed point is a state unchanged by that map for the same input.A non-shared prelude embeds the input, and a non-shared coda maps the final state to next-token logits.Source: §2 · p. 3
  • Convergence is measured by tokenwise relative hidden-state change; a token counts as converged when its change stays below τ = 2% through the observation horizon.The convergence depth is the earliest depth meeting that criterion.Source: §2 · p. 3, Figure 2 · p. 3
  • Tokens converge at different depths, not necessarily in causal order; position correlates only weakly with convergence depth (Spearman correlation 0.14).Figure 2 plots position against convergence depth, with medians over 128-position intervals.Source: §2 · p. 3, Figure 2 · p. 3

3 What Fixed-Point Shaping Enables

Near a fixed point, the endpoint can stand in for the recurrent trajectory. This enables training and inference shortcuts that reduce depth-dependent costs.

3.1 Training: Endpoint Gradients

  • TBPTT differentiates through only the last b recurrences, scaling memory and compute with b rather than R while approaching the fixed-point gradient as states converge.Unlike endpoint gradients applied from initialization, TBPTT is stable early in training and needs no warm-up or estimator switch.Source: §3.1 · p. 4
  • For on-policy RL, saved rollout endpoints let each update reuse one recurrent-block pass and backward computation, making update cost and memory independent of R.Rollout generation still takes most of the training time.Source: §3.1 · p. 5

3.2 Inference: Endpoint Caching and Distillation

  • At training depth R = 5, terminal KV sharing keeps 4 of 12 banks and raises perplexity over the full cache by at most 0.85%.The argument links sharing's low cost to sequence-parallel supervision near fixed points.Source: §3.2 · pp. 5–6
  • A distilled student replaces the recurrent core during prefill by predicting the depth-R endpoint; one teacher recurrence then produces banks close to the teacher's terminal banks.The prelude and coda are reused from the teacher, and decoding remains unchanged.Source: §3.2 · p. 6

4 Improving Fixed-Point Shaping

Fixed-point benefits depend on how training shapes recurrence states. The paper improves depth sampling and input injection, the two techniques it identifies for shaping fixed points.

4.1 Learning a Depth Prior for Fixed-Point Training

  • The learned depth prior adapts supervision to training feedback, balancing prediction quality against fixed-point shaping at equal mean depth.It replaces the fixed PLN prior with a learnable categorical prior over recurrence depths.Source: §4.1 · p. 7
  • The learned prior narrows the gap between fixed-depth training's advantage at R = 5 and PLN sampling's stability across test-time depths.The comparison is on native WikiText-103 test PPL at matched training budgets.Source: Figure 4 · p. 8
  • Entropy regularization spreads depth allocation, trading finite-budget predictive performance for later-depth supervision that supports KV sharing.Increasing the entropy weight widens allocation mainly above the target mean depth.Source: §4.1 · p. 8

4.2 Orthogonal Input Injection

  • Orthogonal injection removes the recurrent state's component along the input direction, preventing that component from amplifying or canceling the injected input.The orthogonal component evolves freely, while the stabilizer makes the input-direction constraint approximate.Source: §4.2 · pp. 8–9

5 Experiments

The experiments evaluate a learned depth prior and OrthoInj, then test rollout-state reuse and distilled prefill as efficiency shortcuts. Models are evaluated across three scales using perplexity and downstream tasks, with most comparisons using terminal KV sharing.

5.1 Setup

  • The models use a one-block prelude, a two-block recurrent core, and a one-block coda, and are evaluated on held-out perplexity and downstream tasks.The three scales are Small, Medium, and Large; unless noted, evaluation uses R = 5 with terminal KV sharing.Source: §5.1 · p. 9

5.2 Learning the Depth Prior

  • Fixed-depth training has the highest perplexity under forced KV sharing and loses over half its GSM8K accuracy at M and L relative to full-cache evaluation.It needs full KV to achieve its best performance.Source: §5.2 · p. 9, Table 2 · p. 10
  • With terminal KV sharing, the learned prior lowers validation PPL by 0.7–1.8% over Fixed PLN-5 and raises downstream average at every scale.At L, it trails Untied 12 by 1.5 downstream-average points while using 3× fewer parameters and KV cache; the entropy term is essential for the L-scale GSM8K result.Source: §5.2 · pp. 9–10, Table 2 · p. 10

5.3 Input Injection

  • OrthoInj has the highest downstream average and lowest validation PPL at every scale among the compared injection schemes.All schemes train with PLN-5 and are evaluated at R = 5 with terminal KV sharing; OrthoInj also ends training with the lowest training loss at every scale.Source: §5.3 · p. 11, Table 3 · p. 11, Figure 5 · p. 11

5.4 Rollout-State Reuse in RL

  • Reuse with Neumann-4 cuts scoring and backward time by 1.99× on GSM8K and 2.02× on MBPP+ versus full BPTT.Reuse trails full BPTT’s pass@1 by no more than run-to-run variation on any benchmark.Source: §5.4 · p. 12, Table 4 · p. 12

5.5 Distilled Prefill

  • Distilled prefill is 1.5–1.8× faster on 8K prompts, while its downstream average trails the teacher by 1.0 points at S, 2.3 at M, and 4.5 at L.At matched latency, it is 0.5–0.9 points higher on downstream average than teacher prefill at R = 2.Source: §5.5 · p. 12, Table 5 · p. 13

6 Related Work

Related work uses recurrence and fixed points for test-time scaling, convergence, and changes to recurrence depth. This paper connects fixed points to efficiency shortcuts and proposes changes to depth sampling and input injection to shape them.

Looped models and fixed points

  • Prior work uses fixed points to stabilize recurrent states, train supervised states as attractors, or halt at convergence; this paper identifies efficiency gains across training, inference, and post-training.Looped models apply shared blocks repeatedly for test-time compute and latent reasoning.Source: §6 · p. 13

Looped model efficiency

  • Truncated backpropagation approximates the equilibrium gradient, while terminal KV sharing is nearly lossless when training supervises near fixed points.Distilled prefill predicts the endpoint with a non-recurrent student, and rollout-state reuse scores RL rollouts from saved states instead of replaying recurrences.Source: §6 · p. 13

Shaping fixed points

  • The paper learns entropy-regularized depth distributions for fixed-point shaping and introduces OrthoInj, which projects the carried-over state off the injected input.Prior depth distributions may be fixed, marginalized at inference, or learned as exits; Parcae improves injection by forcing contractiveness.Source: §6 · p. 14

7 Discussion and Conclusion

The paper concludes that fixed-point endpoints can replace recurrent trajectories across training, inference, and post-training, with outcomes shaped by the depth prior and input injection. It also identifies scale and evaluation boundaries for the reported findings.

7 Discussion and Conclusion

  • A learned depth prior lowers perplexity over a fixed PLN prior while preserving terminal KV sharing, and OrthoInj lowers perplexity relative to Huginn and Parcae injection schemes.Terminal KV sharing and rollout-state reuse reduce memory or time at a small quality cost; distilled prefill trades downstream accuracy for latency, with a gap that grows with scale.Source: §7 · p. 14
  • Models reach at most 1.6B parameters with one seed per configuration; larger scales, other configurations, and real-world feasibility remain untested.The ablations do not exhaust the design space.Source: §7 · p. 14

A Theoretical Analysis and Proofs

  • Appendix A gives the theoretical analysis and proof of Lemma 1, extends it to token-by-token decoding and finite depth, and relates truncated backpropagation to implicit gradients.The appendices also specify methods, training configurations, supplementary experiments, and evaluation protocols.Source: Appendix Contents · p. 19, A Theoretical Analysis and Proofs · p. 20

A.1 Joint Iteration and Frozen-Prefix Decoding

With converged prefix context, joint and frozen-prefix iteration reach the same endpoint under the stated conditions. At equilibrium, truncated backpropagation approximates the implicit gradient, while finite-trajectory and equilibrium objectives can differ.

A.1 Joint Iteration and Frozen-Prefix Decoding

  • Under the stated continuity, invariance, and contraction conditions, joint iteration and sequential decoding converge to the same token states and predictions.The extension applies to one-by-one decoding, which solves and freezes each token in turn.Source: §A.1 · p. 20
  • The median relative endpoint gap is about 4 × 10−7 with a converged prefix, versus 0.27 for Fixed R = 5 and 0.033 for PLN-5 with depth-five prefixes.Qualified prefixes required about a hundred or more recurrences to converge, so practical sharing relies on the finite-depth bound and measured cached perplexity, not exact convergence.Source: Figure 6 · p. 21, §A.1 · p. 21

A.2 Truncated Backpropagation and Implicit Gradients

  • Finite-trajectory BPTT and equilibrium differentiation can yield different gradients even at equal states; before convergence, the Neumann estimate is only a surrogate.A small endpoint residual alone does not guarantee accurate error control at approximate states.Source: §A.2 · pp. 22–23
  • At equilibrium, differentiating only the final recurrent steps gives a truncated estimate of the implicit gradient, with omitted-tail error controlled by contraction.The bound assumes a continuously differentiable recurrence and loss near the equilibrium, with the state Jacobian norm below one.Source: §A.2 · pp. 22–23

B Additional Method Details

  • The appendix details the shared model and training configuration, sampled-depth exposure, learned prior, and injection recipes.The configurations distinguish model scales and variants; each comparison states its scale and configuration.Source: B Additional Method Details · p. 23

B.1 Models and Training Contract

  • Models use separate input and output embeddings, a 64,256-token Jais tokenizer, and bfloat16 arithmetic; peak learning rates are tuned per run.The main text rounds model scales to 100M, 400M, and 1.6B parameters.Source: §B.1 · p. 23, Table 6 · p. 24

B.2 Why Sample Training Depths?

Sampled training depths expose backpropagated updates to converged prefix context, but this exposure alone does not ensure KV sharing. The learned prior uses prediction feedback and entropy regularization to allocate depths.

B.2 Why Sample Training Depths?

  • For Small PLN-5, exposure is 1.10 updates at convergence depth 5 and 0.15 at depth 10, concentrating on prefixes that converge within about ten recurrences.Fixed R = 5 has zero exposure at convergence depths of five or more; the exposure count ignores attention weights and gradient magnitudes.Source: §B.2 · p. 24
  • Exposure is necessary but not sufficient for KV sharing: random-depth loss can leave KV nonstationary, and late-state supervision may not ensure good responses from transient states.The paper therefore measures sharing directly.Source: §B.2 · p. 25

B.3 Learned Depth Prior

  • The learned prior updates depth probabilities using prediction feedback while an entropy term encourages a broader depth allocation.Alternative reward signals replace raw cross-entropy feedback without changing the model CE objective or adding forward passes.Source: §B.3 · p. 25
  • Uniform-prior controls vary both support and initialization: U1:10 restricts depths to 1–10, while U1:64 retains full support and ends at mean depth 6.15.Both controls target mean depth five and use raw-CE feedback.Source: §B.3 · p. 26

B.4 Architecture Variants and Initialization

The appendix specifies alternative injection and normalization recipes, including orthogonal projection of carryover against the injected vector. It also reports a forward-stability issue for a Huginn-Linear variant without exit normalization.

B.4 Architecture Variants and Initialization

  • OrthoInj projects the carryover against the injected vector, while the listed diagonal-injection recipes otherwise use core-entrance injection.The recipes specify the projection anchor and use FP32 projection inner products with ϵ = 10−6.Source: Table 7 · p. 26, §B.4 · p. 26
  • DEQ-QKV injects the fixed prelude output into Q, K, and V through separate bias-free projections shared across recurrent visits.It replaces core-entrance injection and diagonal retention of earlier states.Source: §B.4 · p. 26
  • Huginn-Linear adds a parameter-free core-exit RMSNorm to PreNorm blocks, while the released Parcae readout is omitted and the coda receives the final recurrent state directly.The readout is outside the recurrence and does not enter the fixed-point iteration.Source: §B.4 · pp. 26–27
  • Without exit normalization, S PLN-25 training reached nonfinite loss at all five tested learning rates; adding exit RMSNorm kept replayed inputs finite through recurrence 64.The stability replay diagnoses forward behavior; evaluated Huginn-Linear models were trained from scratch with the exit norm.Source: §B.4 · p. 27

C.1 Depth-Prior Learning and Test-Time Scaling

Learned priors retain task accuracy under terminal KV sharing, unlike fixed-depth training, and entropy changes the trade-off between predictive performance and sharing. Reward and initialization comparisons further distinguish learned-prior variants.

C.1 Depth-Prior Learning and Test-Time Scaling

  • Sharing raises fixed-depth training’s WikiText continuation PPL from 12.19 to 16.71, whereas PLN-5 and all learned priors retain or slightly improve it.At M scale, λH = 0.01 and 0.05 tie for the lowest WikiText PPL at R = 32, 12.93; sharing penalty decreases from +0.96% to −0.22% as entropy rises.Source: §C.1 · pp. 27–28, Figure 7 · p. 28
  • The entropy sweep’s absolute shared MixVal PPL is lowest at λH = 0.01, 4.48, within 0.01 of λH = 0 and 0.05.The comparison reports paired sequence-CE bootstrap intervals as evaluation uncertainty, not training-seed variability.Source: §C.1 · p. 28
  • Learned priors’ terminal sharing changes task-average accuracy by +0.25 to +0.70 points, while Fixed R = 5 loses 0.61 points.The comparison evaluates shared versus FullKV task accuracy at Rpre = Rdec = 5.Source: §C.1 · p. 27, Table 9 · p. 29
  • Inverse perplexity gives the best cached PPL and task mean among S learned-PLN arms; at M, raw CE leads task accuracy while inverse perplexity leads WikiText and cached PPL.The reward variants cost about 1% more FLOPs than fixed PLN; log and exponential CE have higher S perplexity than raw CE or inverse perplexity.Source: §C.1 · p. 30
  • At S, the learned prior’s fixed shape lowers validation PPL by 0.3–1.1% relative to Fixed PLN-5, while adapting that shape during training adds no measurable gain.Fixed priors sampling the learned shape match or beat the learned prior’s validation PPL, with downstream averages within 1.1 points.Source: Table 12 · p. 31, §C.1 · p. 31

C.2 Input Injection

This section evaluates input injection and fixed-point-related approaches to recurrent-model efficiency, including truncated backpropagation, recurrent-state and cache convergence, terminal KV sharing, and distilled prefill. It reports gains from orthogonal injection and terminal sharing, as well as a speed–quality tradeoff in deeper-teacher distillation, with R=5 reported in the main text.

C.2 Input Injection

  • DEQ-QKV breaks terminal KV sharing, reaching CacheEval PPL 16.52 versus 6.43–6.69 for other S schemes; OrthoInj removes carryover alignment that otherwise amplifies the injected input.At the fixed point, Parcae-Decay’s carryover cosine with the injected input is 0.81 (S), 0.87 (M), and 0.91 (L), amplifying input by 2.2×, 2.8×, and 3.4×; projection fixes the factor at one.Source: §C.2 · pp. 31–33, Table 13 · p. 31, Table 16 · p. 33
  • At S, OrthoInj reaches 6.60 validation PPL, and its training CE is lower than Parcae-Decay’s by 0.011 (S), 0.005 (M), and 0.003 (L).The carryover projection and removal of prelude RMSNorm each lower validation PPL, with gains adding up; at M, the downstream average depends on learning rate, with OrthoInj leading by 0.13 points at 10^-3 and trailing by 0.51 at 1.5 × 10^-3.Source: Table 14 · p. 32, §C.2 · p. 32

C.3 Truncation, Depth Sampling, and Fixed Points

  • With depth sampling, M windows of four or more recurrences stay within 1% of full-BPTT WikiText PPL at R = 5, while shorter windows lose 1.6%–15%.At fixed training depth, backpropagating through only the last two recurrences removes degradation beyond training depth and the 21% perplexity loss under terminal-KV sharing, provided input is injected; without injection, truncation erases the input-bearing state.Source: §C.3 · pp. 33–34, Table 17 · p. 34, Table 18 · p. 35

C.4 Recurrent-State and Cache Dynamics

  • More than 97% of tokens meet the 1% hidden-state-change criterion within 64 updates, while five-recurrence KV change falls from 1.23% at r = 25 to 0.59% at r = 160.The KV norm remains at 97.48% of its r = 5 value; these measurements characterize settling, not global contraction or a causal training-prior effect.Source: Table 19 · p. 35, §C.4 · pp. 35–36

C.5 Terminal KV Sharing at Deeper Recurrence

  • Terminal KV sharing raises MixVal PPL by at most 0.85% at R = 5 and 0.11% at R = 16 and 32, while keeping GSM8K accuracy within 3.2 points of the full cache.At M scale, sharing shifts the seven-task average by at most 0.34 points at R = 5 and 0.04 points at R = 16.Source: §C.5 · p. 36, Table 20 · p. 36, Table 21 · p. 37

C.6 Independent Prefill and Decode Budgets

  • For mean-25 Small checkpoints at Rdec = 2, Rpre = 1 gives cached log-perplexity 2.89 versus 3.13 for Rpre ≥5, while raising Rdec from 1 to 10 lowers it from 3.47 to 1.82.Beyond Rpre = 5, cached log-perplexity changes by at most 0.01 at each decode depth once decode depth is at least 5 (or 2 for the mean-5 model).Source: §C.6 · p. 37, Figure 11 · p. 37

C.7.1 Teacher-Initialized Students under the Full Battery

  • Teacher initialization with KL raises the downstream average from 45.66 to 46.77 on Medium and from 39.96 to 40.63 on Small.On Medium, adding KL raises GSM8K from 8.42 to 8.95, DROP from 15.21 to 16.20, and MBPP+ from 5.38 to 6.12; neither standalone student nor KV-direct use beats Untied 4.Source: §C.7.1 · p. 38, Table 22 · p. 39

C.7.2 A Deeper Teacher at R = 25

  • At R = 25, distilled prefill is 4.3× faster than the teacher on Small and 5.4× on Medium, but the student trails its teacher by 2.4 and 3.9 downstream points, respectively.On Medium, teacher initialization raises the downstream average from 42.09 to 45.54; the R = 25 student is slower and less accurate than the R = 5 student at each scale.Source: §C.7.2 · p. 39
  • The R=25 teacher saves more prefill time but loses more quality than the R=5 teacher.The comparison uses Mean-25 learned-prior teachers at R=25 alongside the R=5 teachers of Table 5; AVG averages seven choice tasks.Source: Table 23 · p. 40
  • The deeper teacher’s decoder appears more sensitive to small handoff errors, so a stronger teacher need not yield a better handoff [56] [57].The authors therefore report R=5 in the main text.Source: §C.7.2 · p. 40

C.8 Reusing States during Post-Training

Rollout-state reuse estimates RL updates from saved endpoints, reducing scoring and backward time while retaining pass@1 near the comparison methods. The section also reports code-RL results and bounds the evidence with estimator and evaluation-scope caveats.

C.8 Reusing States during Post-Training

  • The estimator records one application of the recurrent map at the saved endpoint and uses Neumann products; its approximation has no convergence guarantee.Neumann-4 uses four state products, while full BPTT differentiates all six recurrent visits in this experiment.Source: §C.8 · p. 40
  • Reuse + Neumann-4 gains 21.03 pass@1 points over initialization and differs from full BPTT by −1.55 points, with a 95% interval of [−3.13, 0.05].Reuse and recomputation have gradient cosine 0.997–0.999; the reported +1.60-point difference between them is consistent with run-to-run variation, and intervals exclude seed variance.Source: §C.8 · p. 41
  • At R = 6, the average relative state residual is 0.034, larger than near a converged fixed point, so the finite Neumann series truncates more of the full gradient.The reported maximum residual is 0.041; the experiment does not test how savings scale with depth because four state products are close to the six full-BPTT visits.Source: §C.8 · pp. 40, 42
  • Multiple training seeds, the full test split, and reuse across optimizer updates remain untested; saved states are rejected when missing, stale, or token-mismatched.Saved states are used only before an optimizer step changes the parameters.Source: §C.8 · pp. 40, 42
  • On MBPP+, all methods raise pass@1 from 30.9% to 37–41%, while Reuse + Neumann-4 ends five points below full BPTT on pass@8 (64.0 vs. 69.0).The comparison uses one pass over 272 training tasks and evaluation on 100 held-out tasks.Source: §C.8 · p. 42, Table 26 · p. 43
  • 2.02× over full BPTT and 1.27× over recomputation, reuse speeds scoring and backward computation, but not end-to-end training because rollout time dominates.Rollout generation takes 72–82% of training time, and reuse rollouts take longer than full BPTT rollouts in these runs.Source: §C.8 · p. 43

D Datasets and Evaluation Protocols

  • Evaluation paths were checked against references on subsets, but results generally use one training seed and intervals quantify evaluation-sample uncertainty.Pretraining decontamination was not verified; because arms share a checkpoint, contamination could shift absolute accuracy without changing their comparison.Source: §C.8 · p. 41, D Datasets and Evaluation Protocols · p. 43

D.1 Mixture-Matched Validation

  • MixVal contains 9,961,472 next-token targets from 64 packed 8,192-token sequences drawn from each of 19 TxT360 validation sources.Its PPL is exp(CE) from token-weighted teacher-forced cross-entropy; document-level separation from training remains unverified.Source: §D.1 · p. 43

D.2 WikiText-103 and Zero-Shot Tasks

This section specifies the paper’s evaluation tasks, scoring procedures, and terminal-KV protocols. It also places the paper’s methods in relation to extended related work.

D.2 WikiText-103 and Zero-Shot Tasks

  • WikiText-103 evaluation concatenates 2,891 nonempty records and scores 310,315 next-token targets in 38 windows of 8,192 tokens at depths 1–64.Paired cached PPL scores 271,403 continuation targets after a 1,024-token prefill, with terminal sharing reducing twelve banks to four.Source: §D.2 · p. 43
  • The zero-shot battery includes six Mamba-3 suite tasks plus SciQ; its seven-task AVG equally weights accuracies and excludes PPL and generation.Choice-task examples use complete labeled splits, and the average includes SciQ at R = 5.Source: §D.2 · pp. 43–44

D.3 Terminal-KV Evaluation

  • At R = 5, terminal-KV evaluation stores one bank per physical attention layer, retaining each core layer’s final-recurrence KV for earlier tokens.The empty-cache pass matches the native forward pass and differs only in what it stores; checkpoints passed this check on MixVal prefixes.Source: §D.3 · pp. 44–45

D.4 Cached-Continuation Perplexity

  • Cached-Continuation PPL uses teacher-forced incremental decoding: prefill logits score the first continuation target, then each appended reference token scores the next.Predicted EOS tokens do not stop scoring.Source: §D.4 · p. 45
  • CacheEval uses 304 frozen MixVal spans, each with 1,024 prompt tokens and 512 continuation targets; prior, injection, and depth comparisons use five recurrences.The cache grows from 1,024 to 1,535 tokens over 511 teacher-forced inputs.Source: §D.4 · p. 45

D.5 Generation Benchmarks

  • Generation evaluation uses greedy DROP and GSM8K scoring, while MBPP+ estimates pass@1 from eight samples per task at temperature 0.80.The stated test sizes are 500 frozen DROP examples, all 1,319 GSM8K test examples, and 100 frozen MBPP+ tasks.Source: §D.5 · p. 45

E Extended Related Work

  • The extended related-work discussion situates the learned depth prior, OrthoInj, terminal-KV sharing, rollout-state reuse, and distilled prefill against prior work.These methods receive only brief treatment in the main related-work section.Source: E Extended Related Work · p. 45

E.1 Learned Depth Distributions and Exits

The section presents an unconditional, entropy-regularized training depth prior and orthogonal input injection, then relates fixed-point reuse and distillation to prior methods. It also distinguishes cache-sharing compatibility across models and the roles of rollout-state reuse and distilled prefill.

E.1 Learned Depth Distributions and Exits

  • Unlike RL-Halting, the unconditional training prior samples one depth per microbatch, starts from capped PLN, and entropy-extends supervision above expected depth for terminal KV sharing.RL-Halting unrolls to maximum depth to form its state-dependent distribution, while the prior runs only the sampled depth.Source: §E.1 · p. 46
  • An exit may stop once prediction suffices, so training need not reach depths beyond prefix convergence where supervision prepares the model for terminal KV sharing.A single exit distribution also determines both inference stopping and per-depth training-loss weights.Source: §E.1 · p. 46

E.2 Input Injection and Orthogonal Updates

  • OrthoInj projects the carried-over state off the injected input, fixing its component along that input while the orthogonal component evolves; prior schemes inject at different points.DEQ-Transformer injects into Q/K/V, DEQ-TrellisNet before convolutional LSTM gates, and Huginn and Parcae at the recurrent-core entrance.Source: §E.2 · p. 46

E.3 KV Sharing Across Recurrent Depth

  • In common GSM8K without retraining, one shared KV slot per layer leaves Huginn nearly unchanged at 32 recurrences but collapses Ouro at four; Huginn long-form behavior remains unmeasured [9].The cited studies test sharing zero-shot or after adaptation, while the authors change only the training depth policy and do not measure long-form reasoning.Source: §E.3 · p. 47

E.4 Fixed-Point Reuse

  • Rollout-state reuse scores sampled tokens from rollout-saved endpoints and backpropagates through one recurrence per endpoint, avoiding replay during the RL update.Unlike reusing a fixed point to initialize a new forward solve, it carries states between rollout and policy-update passes over the same tokens.Source: §E.4 · p. 47

E.5 Distillation for Looped Models

  • Unlike recurrent-student distillation, distilled prefill uses a separate non-recurrent student to predict a frozen teacher’s endpoint for prompts; decoding retains the teacher’s recurrence.One teacher recurrence from the student prediction builds the KV banks used under terminal sharing.Source: §E.5 · p. 47
Loading 2610.06833v1…