BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

• Prior KV-sharing results were inconsistent: some models reused the final recurrence’s cache with minimal quality impact, while other recent work reported significant degradation.Even models trained under identical setups did not consistently exhibit KV-shareability.Source: §1 · pp. 1–2
• Stochastic-depth pretraining induces contractive residual dynamics, and BeNice! corrects first-recurrence residual errors by jointly optimizing shared first-recurrence and unshared final-recurrence predictions.BeNice! injects the final recurrence’s KV-cache into the first recurrence.Source: §1 · p. 2

End-to-End Quality

• BeNice! improves shared Core and Core-extended quality by up to 8.35 and 4.56 points, respectively, while matching standard finetuning’s unshared validation perplexity.Compared with standard finetuning, BeNice! improves validation perplexity by 9.75 points when sharing.Source: §1 · p. 2

Complementary to other KV Compression Methods

• KV-sharing composes with MLA and KDA sequence mixers and with low-precision KV-cache quantization, including fp8, mxfp4, and nvfp4.Source: §1 · p. 2

Accelerating Inference Workloads

• Under simulated long-context inference traffic, KV-sharing improves cache-hit rates by up to 31.4% (8.1 points), increases throughput by up to 23.6%, and reduces p90 TTFT by up to 83.6%.Source: §1 · pp. 2–3

2 Background

• Prior looped language models vary in recurrence placement, halting, and gradient computation; this work studies middle-looped models with implicit halting and explicit differentiation.Looped architectures have also been explored in synthetic problem-solving tasks.Source: §2 · p. 3

2.1 Prior Looped Models

• Middle-looped models typically apply a prelude, recurrent block, and coda sequentially, with the recurrent block refining a hidden state across recurrences.Source: §2.1 · p. 3
• Prior models either sample recurrence depth during training or use adaptive computation time, and some use truncated backpropagation through time to bound training memory.Parcae and Huginn sample depth, while Ouro uses a learned router to decide when to halt.Source: §2.1 · p. 3

2.2 Inference Overhead of Looped Language Models

• Inference memory includes model weights and KV-caches; for looped models, conventional caching makes KV memory grow linearly with recurrence count while parameter memory stays fixed.The passage gives KV-cache memory as O(LD d_kv) and model-weight memory as O(P).Source: §2.2 · p. 3
• Table 1 compares looped-model quality with and without a shared KV-cache and indicates that KV-shareability is not universal.Source: Table 1 · p. 4

3 Understanding KV-shareability in Looped Models

• The study evaluates shared inference, which reuses the final-recurrence cache across recurrences, against unshared inference, which retains each recurrence’s cache.Models are trained from scratch at four scales, varying whether recurrence depth is sampled or fixed during training; KV-shareability is not explicitly trained for.Source: §3 · p. 4, §3.1 · p. 4

3.2 KV-Sharing Fails to Converge to a Common Fixed Point

• Stochastic depth alone does not guarantee shared and unshared fixed-point agreement: KV-sharing can cause token-independent residual errors that shift the fixed point and harm quality.The failure is localized to the first layer of the first recurrence; correcting its mean projected error can remove the need to cache that layer separately.Source: §3 · p. 4, §3.2 · pp. 6–7

3.1 Stochastic Depth Enables Contractive Residual Dynamics

• KV-cache convergence supports shareability because replacing an intermediate cache with a fixed-point cache perturbs attention output by a term bounded by the fixed-point residual.A contraction drives hidden states toward a fixed point, and locally Lipschitz key and value projections make their caches converge as the residual steps shrink.Source: §3.1 · pp. 4–5
• Stochastic-depth training induces fixed-point convergence and coincides with the emergence of KV-shareability, whereas static-depth training lacks that incentive and performs worse under sharing.The final-recurrence cache is empirically the optimal broadcast target because it lies closest to the fixed point across recurrences.Source: §3.1 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5

3.2 KV-Sharing Fails to Converge to a Common Fixed Point

• KV-shareable models converge to similar fixed points, with mean relative differences at most 1.69%, while non-shareable models reach up to 177.5%.The comparison covers Parcae 140M and 370M versus 770M and 1.3B.Source: §3.2 · p. 6

4 Learning to Share Across Recurrences

• The two-stage method uses stochastic-depth pretraining to induce contractive dynamics, then BeNice! to correct the token-independent basis shift by optimizing the reused-cache first recurrence alongside the standard final recurrence.Direct training on sequential KV-sharing rollouts is described as computationally infeasible and slow.Source: §4 · p. 7

4.1 Stochastic-Depth Pretraining: Enabling Contractive Recurrent Dynamics

• Per-sequence stochastic-depth pretraining samples recurrence depth from a Poisson distribution and uses truncated backpropagation to keep pretraining memory bounded.The sampled depths drive KV-caches toward a common shareable limit across recurrences.Source: §4.1 · p. 7

4.2 BeNice! Post-training: Correcting First Recurrence Residual Errors

• BeNice! trains first-recurrence queries and residuals using final-recurrence KV, while its auxiliary loss promotes compatibility and its standard language-modeling term maintains later-depth quality.Finetuning samples recurrence depth from the same distribution used during pretraining.Source: §4.2 · p. 7
• BeNice! extends to multi-latent attention [13] and Kimi delta attention [65] by swapping the final recurrence’s latent or hidden state, respectively, over the KV-cache.Source: §4.2 · p. 8

5.2 KV-Shareability is Complementary to Other KV Compression Methods.

• KV-sharing with BeNice! changes Core and Core-Extended by only 0.09 and 0.04 points on average across sequence mixers, respectively.BeNice! enables KV-shareability with MHA, MLA, and KDA; the figure reports the mixer comparison.Source: Figure 6 · p. 9, §5.2 · p. 10

5.1 Enabling KV-Shareability while Maintaining End-to-End Quality

• Up to 8.35 and 4.56 points, BeNice! improves shared Core and Core-Extended quality, respectively, while matching standard finetuning quality without sharing.KV-sharing reduces validation perplexity by up to 45.2% (9.75 PPL points) versus standard finetuning when caches are reused.Source: §5.1 · pp. 8–9, Table 5 · p. 9

5.2 KV-Shareability is Complementary to Other KV Compression Methods.

• Across all model scales and sequence mixers, KV-sharing decreases Core and Core-Extended by 0.09 and 0.04 points on average, respectively.KV-sharing and quantization interact by at most 0.53% in ClimbMix perplexity, supporting their reported complementary compression behavior.Source: §5.2 · p. 10
• KV-sharing with quantization changes Core and Core-Extended by −0.02 and +0.06 points on average, and increases ClimbMix validation perplexity by 0.67% on average.The reported interaction between KV-sharing and quantization is at most 0.53% in ClimbMix perplexity.Source: §5.2 · p. 10, Figure 7 · p. 10

5.3 Saving Memory and Accelerating Inference Workloads with KV-Shareability

• At batch size 64 and context length 2048, KV-sharing reduces global HBM usage by 66.2%, from 63.82 GB to 21.60 GB.At μrec = 8, it raises token capacity to 3.33× the unshared capacity under the same KV-memory budget.Source: Figure 8 · p. 11, §5.3 · p. 11
• At 2 req/s, KV-sharing improves throughput by 23.6% and reduces p90 TTFT from 139.2 s to 23.2 s.At low loads such as 0.5 req/s, it matches the unshared baseline in TTFT and throughput; Figure 10 also reports concurrency-frontier comparisons.Source: Figure 10 · p. 11, §5.3 · p. 11

6 Conclusion

• The paper attributes KV-shareability to contractive residual dynamics that drive KV-caches toward a common limit, and proposes stochastic-depth pretraining followed by BeNice! to correct first-recurrence residual errors.The conclusion says the method composes with memory-efficient sequence mixers and KV-quantization, detaches inference memory overhead from test-time recurrences, and improves throughput and TTFT in memory-constrained settings.Source: §6 · pp. 11–12

A Glossary

• Table 6 collects notation and terminology, including tensor dimensions, Parcae blocks, tensors and distributions, and KV-cache compression definitions.The table is organized into those three groups.Source: A Glossary · p. 20, Table 6 · p. 20

B Extended Literature Review

• The literature review distinguishes middle-looped models, fully looped models, and KV-sharing approaches, including variants with first-recurrence or sliding-window caches.It cites Bae et al. for a first-recurrence sharing variant and Vendrell et al. [69] for first-recurrence sharing alongside per-recurrence sliding-window caches.Source: B Extended Literature Review · pp. 20–21

C Evaluation Setup of KV-Shareability

• The evaluation tests KV-shareability with perplexity and question-answering benchmarks; the listed tasks include a 22-task CORE subset and 46-task CORE-Extended suite.The QA protocol prefills prompts using the final-recurrence cache and scores candidate continuations against it; the PPL protocol streams tokens using final-recurrence keys and values.Source: C Evaluation Setup of KV-Shareability · pp. 21–22, Table 7 · p. 23
• For the shared-cache perplexity evaluation, the final recurrence’s keys and values are broadcast across all recurrences, requiring token-by-token prefix reprocessing and O(T |s|^2) per-sequence cost.The unshared baseline uses a standard parallel teacher-forced forward pass, with O(T |s|) cost.Source: C Evaluation Setup of KV-Shareability · pp. 21–22

D Proof of KV Convergence

• If the recurrent block is a contraction and the per-layer key/value projections are Lipschitz, hidden-state convergence transfers to per-layer key/value convergence at a rate controlled by the step-to-step residual.The proof states that contraction gives a unique fixed point and convergence to it.Source: D Proof of KV Convergence · p. 24
• The proof does not guarantee that Parcae or Huginn is contractive, and assumes Lipschitz key/value projections rather than globally Lipschitz self-attention.The passage notes prior empirical fixed-point observations and separate verification, while limiting the proof to the projection assumption.Source: D Proof of KV Convergence · p. 24

E KV-Sharing Error Subspace

• The KV-sharing error variance is low-rank and is well explained by 8 right singular vectors, defining the KV-sharing error subspace.The error is formed from per-token differences between first-recurrence attention outputs using that recurrence’s KV cache versus the shared final-recurrence cache.Source: E KV-Sharing Error Subspace · pp. 24–25
• Naive Patching adds the average projected subspace error only to the first layer’s first-recurrence attention output and is nearly lossless for KV-sharing quality.Other layers and later recurrences receive no patch.Source: E KV-Sharing Error Subspace · p. 25

F Parcae and Huginn Converge to Fixed-Points

• Residual states and attention patterns stabilize with recurrence depth in released Parcae and Huginn models, with Parcae 1.3B converging more slowly than Parcae 140M–770M.The models were trained with stochastic depth, and the fixed-point dynamics are present in the original pretrained models.Source: Results · p. 26, Figure 12 · p. 27

Stochastic and Static Depth Training

• The comparison trains size-matched Parcae models with stochastic or static recurrence depth and evaluates residual convergence, attention convergence, and KV-sharing quality.Static-depth training uses T = 8; stochastic depth uses a Poisson distribution, with µrec = 8 and µbwd = 4.Source: G Expanded Results Comparing… · p. 28
• KV-sharing is nearly lossless for stochastic-depth models at every scale, while static-depth models degrade substantially under compression.The two training approaches have similar training-loss trajectories and comparable uncompressed quality.Source: G Expanded Results Comparing… · p. 28, Figure 13 · p. 28, Table 8 · p. 29

H Extended Empirical Analysis on KV-Sharing Fixed Point Convergence Failures

• Parcae 140M and 370M have 0.82% and 1.69% mean relative fixed-point differences, while Parcae 770M and 1.3B have 134.8% and 177.5%, respectively.The larger models converge to different points with and without KV-sharing; after BeNice! finetuning, trained baseline models converge to the correct fixed point.Source: H Extended Empirical Analysis on… · pp. 29–30

I Extended Finetuning Setup

• BeNice! finetuning uses stochastic recurrence-depth sampling with µrec = 8 and is evaluated across three finetuning runs and three evaluation seeds.The extended setup applies BeNice! to released and newly trained Parcae models.Source: I Extended Finetuning Setup · p. 30

J LoRA Finetuning for KV-sharing

• Both full BeNice! and BeNice!†, its LoRA finetuning version, enable KV-sharing while maintaining quality across benchmarks.BeNice!† uses rank-8 adapters on first-layer key and value weights while freezing the rest of the model.Source: J LoRA Finetuning for KV-sharing · pp. 30–31, Table 11 · p. 31

K Extended Discussion of Experimental Setup for MLA and KDA Evaluation

• For MLA, KV-sharing fails as model size increases and mean patching fails to fully recover quality at 770M, whereas KDA models have KV-sharing out of the box.The MLA scheme retains only the final recurrence’s latent cache; KDA retains only the final recurrence’s sequence-mixing hidden state.Source: K Extended Discussion of Experimental… · pp. 31–32, Table 12 · p. 32

L Extended Compression and Patching Results for Newly Pretrained Models

• All four MHA models have KV-sharing errors; retaining first-layer KVs from the first recurrence alongside normal sharing preserves lossless quality except at 1.3B, where BeNice! yields near-lossless quality.Mean patching fails to recover quality for the 770M MLA model, while KDA variants perform well on KV-sharing out of the box.Source: L Extended Compression and Patching… · p. 32

M Optimal Hyperparameter Transferability Across Recurrence

• Optimal learning rate and batch size transfer strongly across recurrence for all model classes, as do Muon momentum and warmup ratio with minor variation in flat regions.Lower-recurrence sweeps can therefore reduce the FLOPs needed to find hyperparameters that generally remain optimal at higher recurrence; lower Spearman correlations coincide with flatter, noisier landscapes.Source: M Optimal Hyperparameter… · pp. 33–34, Figure 14 · p. 33, Figure 15 · p. 33

N Model Definitions and Configurations

• The stochastic-versus-static comparison uses a 1:7 parameter-to-token ratio, with static-depth models held deterministically at µrec = 8 recurrences.The pretraining models use the configurations and hyperparameters specified for the paper’s experiments.Source: N Model Definitions and Configurations · p. 35

Pretraining

• Pretraining hyperparameters are listed in Table 16, except for stochastic and static recurrence-depth models, which are addressed in Section 3 and explained in Section G.Table 16 covers climbmix-trained Parcae models, including MLA and KDA variants.Source: O Hyperparameters and Training Details · p. 35, Table 16 · p. 37

BeNice! and Standard Finetuning

• BeNice! performs best across models at a learning rate of 1.25e−4 and γ of 0.01, while Standard Finetuning favors 2e−4 for 770M models and 8e−5 for 1.3B models.These settings are selected on a held-out validation set; the sweeps use a 500-step optimizer budget, with final learning rates rescaled for 1800 steps.Source: O Hyperparameters and Training Details · p. 35

Reference tables

• The appendix tables cover baseline Parcae and its MLA and KDA variants, Parcae optimizer parameter-group assignments, and pretraining hyperparameters.The table captions identify these scopes without specifying their contents here.Source: Table 14 · p. 36, Table 15 · p. 36, Table 16 · p. 37

BeNice!† Finetuning

• BeNice!† uses a limited sweep on released Parcae 770M and 1.3B models; both perform best at 1e−3 and γ of 0.06.The sweep uses the full 1000-step optimizer budget, and the method uses a rank 8 adapter with α = 16.Source: O Hyperparameters and Training Details · p. 37