Recirculation

State tracking

• Tracking deficits can contribute to ineffective working memory, unstable theory-of-mind representations, and the absence of a robust internal world model (Song et al., 2026).Other reported consequences include conversational incoherence (Laban et al., 2025), inefficient information gathering, and impaired multi-agent communication and cooperation (Davidson et al., 2025; Khatua et al., 2026).Source: §1 · pp. 1–2
• Contextualization errors arise when models reverse a committed interpretation without acknowledging the inconsistency (Lepori et al., 2025).Maintaining multiple possible meanings is reasonable, but switching from a previously committed meaning should be acknowledged. As belief distributions become prohibitively high-dimensional, approximations such as sampling, prototypes, and concrete mental models still require state tracking.Source: §1 · p. 2
• Shallow transformer layers cannot access deeper contextualized representations, allowing response generation to outpace semantic convergence (Lepori et al., 2025).Parallel training and prefill preclude traditional sequential state updates (Mozer et al., 2026), although transformers learn finite-length workarounds (Piotrowski et al., 2025; Prakash et al., 2026; Shai et al., 2024). In the bank example, early response processing can rely on an ambiguous representation despite deeper layers having resolved its meaning.Source: §1 · pp. 1–2, Figure 2 · p. 3
• 60% fewer contextualization errors followed copying a disambiguated deep-layer representation into a shallow layer at a critical token (Lepori et al., 2025).The experiment used generated questions requiring contextualization of a critical token, with activation patching restricted to that token.Source: §1 · p. 2
• Leaking small amounts of deep-layer activation into shallow layers at every token might enrich representations without retraining or pushing them out of distribution.The proposal contrasts weak feedback with wholesale representation replacement, which the authors conjecture would cause harmful feedback amplification if applied indiscriminately.Source: §1 · p. 3
• Residual-stream alignment could support direct communication between deep and shallow layers without an arbitrary adapter.The argument assumes corresponding features across layers: additive residual contributions have identical direct effects on output regardless of their depth. Feeding back a disambiguated representation could therefore amplify useful semantic features for subsequent processing.Source: §1 · p. 3

2 Proposed method: Recirculation

• Recirculation introduces recurrence across both depth and input steps, allowing successive states to occupy the same layer rather than progressively deeper layers as in looping.After warm-up, two input stacks run in parallel, with readout following a stack’s first iteration. All reported experiments use one additional iteration; unbounded iterations would make recirculation a true recurrent neural network.Source: §2 · p. 4, Figure 3 · p. 5
• Recirculation requires sequential passes even during prefill, whereas looped transformers can process a fixed input sequence in parallel.Looped transformers repeatedly apply shared blocks within each input stack, freezing or caching earlier stacks during autoregressive decoding. Recirculation’s cross-step state updates prevent equivalent sequence-wide parallelization.Source: §2 · p. 4
• Recirculation uses a convex mixture of deep-source and shallow-destination activations, unlike looping, which replaces the incoming residual stream with the looped activation.Unless otherwise stated, the mixture coefficients satisfy β = 1 − α, and the source is rescaled to match the destination’s L2 norm to control activation magnitudes.Source: §2 · pp. 3–4

Looped transformers

• Looping increases depth without adding parameters but does not guarantee indefinite state tracking; its inference-time variants most closely resemble recirculation.Looping reruns layers (Giannou et al., 2023) and can increase expressivity (Saunshi et al., 2025), but differs conceptually from recirculation’s recurrence. Related approaches use pretraining, fine-tuning (Koishekenov et al., 2025; McLeish et al., 2025), or inference-only reasoning improvements (Li et al., 2025b; Chen et al., 2026).Source: §3 · pp. 5–6

Training objectives

• Training objectives encourage stateful layer embeddings interpretable as state-update functions (Hu et al., 2025; Teoh et al., 2025; Huang et al., 2026).These losses target exact or approximate state-updating solutions where such solutions exist.Source: §3 · p. 6

State tracking

• Transformer state tracking is constrained by model depth and increasingly difficult use of state representations as they move into deeper layers.Merrill and Sabharwal (2025) prove log n layers necessary and sufficient for recognizing regular-language strings of length up to n and graph connectivity with n vertices, establishing constructability rather than learnability. Trained depth-limited models can nevertheless solve specific finite-length problems (Piotrowski et al., 2025; Prakash et al., 2026; Shai et al., 2024).Source: §3 · p. 6

Recurrent transformers

• Sequential recurrent updates can express arbitrary state dynamics, while many state-space models are no more expressive than ordinary transformers (Merrill et al., 2025).Recurrent transformers may update token-by-token, but most use blockwise recurrence. The passage distinguishes these capabilities from state-space models, stating that none are as expressive as a generic recurrent network.Source: §3 · p. 6

Thinking models

• Chain-of-thought thinking propagates state from deep to shallow layers, increasing expressivity through recurrence (Li et al., 2024; Merrill and Sabharwal, 2024).Thinking can use natural-language tokens or latent space (e.g., Hao et al., 2025; Jolicoeur-Martineau, 2025). Training can be costly because this recurrence restricts parallelism.Source: §3 · p. 6

Activation steering

• Recirculation provides inference-time self-steering, using contextualized deep-layer activations to guide shallower representations rather than static, externally derived steering vectors.Activation steering modulates behavior through latent-representation interventions (Turner et al., 2023; Zou et al., 2023; Gao et al., 2025). These representations encode directions governing truthfulness and refusal (Marks and Tegmark, 2023; Arditi et al., 2024).Source: §3 · p. 6

4.1 Hyperparameter sweeps; 4.2 Perplexity evaluation

• Up to 16% perplexity reductions for Gemma3 1B/4B and 35% for 12B accompany robust improvements on nine of ten datasets versus no recirculation.Sweeps selected source/destination pairs {11, 4}, {18, 9}, and {35, 16} for 1B, 4B, and 12B; increasing α amplified effects but made more pairs harmful, while evaluation used α = 0.15 on separate evaluation splits. Lambada was anomalous because of short sequences and tokenization artifacts; the 12B model is relatively weak at language modeling despite strong instruction-tuned downstream performance (Gemma Team et al., 2024).Source: §4.2 · p. 7, §4.1 · p. 7, Table 1 · p. 8

4.3 Normalization and ramping

• Source-embedding renormalization stabilizes hyperparameter sweeps, while early-token ramping slightly reduces Gemma3 1B perplexity and larger models require non-convex mixing.Renormalization improves robustness across source/destination pairs rather than maximum perplexity reduction; Gemma3 4B and 12B critically require β = 1 instead of β = 1 − α. Attenuating α for initial tokens addresses early-token harm in 1B, which was not observed in 4B or 12B.Source: §4.3 · p. 8

4.4.1 Robustness across model families; 4.4.3 Recirculation versus looping

• About 5% versus less than 0.5% perplexity reductions distinguish Gemma3 from four other receptive model families; training-free looping lacks similarly robust Gemma3 benefits.Ministral3, Pythia, Qwen3, and Phi2 benefit most in middle layers, but their α and normalization were not tuned; Gemma2 and Gemma4 show gains as pronounced as Gemma3. Unlike recirculation’s cross-scale benefits, training-free looping shows benefits only at larger Gemma3 scales.Source: Figure 7 · p. 9, §4.4.1 · p. 9, Figure 8 · p. 10

4.4.2 Recirculation versus temperature tuning

• 19.55% perplexity reduction combines recirculation with temperature tuning on Gemma3 1B, versus 14.21% for recirculation alone and 8.48% for temperature tuning alone.Both temperature-tuned conditions use temperature 1.2 rather than the default 1.0; the nearly additive gains rule out token-independent temperature adjustment as the explanation for recirculation.Source: §4.4.2 · pp. 9–10

4.4.4 Which tokens benefit from recirculation?

• Recirculation benefits depend on token position and content, with strong short-lag effects, persistent long-lag gains, and harm at the earliest positions.In Gemma3 1B’s 1024-token windows, roughly t < 10 is harmful, while positions 20–200 retain benefits at lag 256; adverbs, adjectives, and verbs benefit most, whereas numerals, determiners, and pronouns benefit least. Plural nouns benefit robustly but singular nouns do not, a distinction also noted by Galashov et al. (2025); complementary all-but-one experiments suggest additive log-likelihood gains supporting persistent state representations.Source: §4.4.4 · pp. 10–11, Figure 9 · p. 11

4.5.1 Instruction following; 4.5.2 Contextualization

• About 25% and 75% error-rate reductions improve Gemma3 4B and 12B IT instruction following, respectively, using perplexity-selected recirculation hyperparameters.Task-specific layer sweeps further improved accuracy but deliberately overfit the instruction-following task. Contextualization improved two of three question types for 1B and 4B but hurt two for 12B; hyperparameters came from pretrained models, and instruction-tuned sweeps produced larger gains.Source: §4.5.1 · pp. 11–12, Figure 10 · p. 12, §4.5.2 · p. 12

4.5 Generative tasks; 4.5.3 Multiple-choice and single-token response tasks; 4.5.4 GSM8k

• Six of eight single-token benchmarks improve modestly with recirculation over the Gemma3 4B PT baseline, while GSM8k improves at both pass@1 and pass@128.Zero-shot single-token evaluations used MMLU development-set hyperparameter selection and showed modest positive and negative accuracy differences. Zero-shot GSM8k improvements indicate both capability sharpening and expansion, with greater apparent benefit for extended responses than single-token responses.Source: Table 2 · p. 13, §4.5.3 · p. 13, Figure 12 · p. 14

4.6 Adaptive recirculation

• Adaptive recirculation’s token-conditioned mixing vectors outperform scalar or static alternatives and slightly exceed full fine-tuning for Gemma3 1B perplexity reduction.With source and destination layers fixed, an MLP learns vector-valued α and β from token-specific source and destination embeddings. Training uses 250 documents each from arXiv, C4, and PG19; six variants are evaluated across nine datasets.Source: Figure 13 · p. 14, §4.6 · p. 15
• 23.0% mean perplexity reduction with adaptive recirculation exceeds basic recirculation’s 8.5% reduction and full Gemma3 1B fine-tuning’s 21.6% reduction.Adaptive recirculation beats basic recirculation on all nine datasets. It leaves Gemma3 unchanged, reducing the stated risk of overfitting through model fine-tuning.Source: §4.6 · p. 15
• Adaptive recirculation’s single-token task results depend critically on training data: MMLU adaptation improves performance, whereas ARC Easy or ARC Challenge adaptation produces significant drops.Adapting to 3,600 of about 14,000 MMLU test examples yields across-the-board improvements, but MMLU evaluation includes training examples; improvements on other datasets demonstrate generalization. The auxiliary MMLU training split yields improvements on balance but is lower quality and potentially contaminated (Hendrycks et al., 2021); the ARC training datasets are much smaller.Source: §4.6 · p. 15
• 8.8% and 20.9% reductions in GSM8k error rate are achieved by adaptive recirculation with pass@1 and pass@128, respectively, without changing model weights.These gains support its promise for extended generative responses despite the lack of robust improvements on single-token response tasks.Source: §4.6 · p. 16

5 Discussion

• Training-free recirculation reduces perplexity and improves instruction following, question answering, and problem solving in pretrained and instruction-tuned Gemma3 models.Generation adds almost no computation cost because modern hardware efficiently parallelizes the two transformer stacks; autoregressive prefill can nevertheless be slow for large contexts.Source: §5 · p. 16
• Recirculation shares residual-pathway representation alignment with training-free looping but remains distinct and complementary to looping, variable computation time, latent thought, and chain of thought.The discussion identifies training-free looping (Li et al., 2025b; Chen et al., 2026) as an exception to predominantly training-based architectural research and attributes representation alignment to Elhage et al. (2021).Source: §5 · p. 16

5.1 Model-design affordances

• Recirculation explores model-design affordances: using pretrained models’ internal properties to guide architectural changes rather than proposing modifications that require costly training.The model informs recurrence placement, activation mixing, and normalization; observations suggest simple activation-vector mixing may suffice instead of adapters or cross attention. Adaptive experiments support the hypothesis that useful weight-preserving modifications reveal inductive biases that could facilitate training.Source: §5.1 · p. 16

5.2 Limitations and future directions

• Task-dependent hyperparameters constrain practical applicability unless universal settings or an automatic task-to-hyperparameter mapping can be identified.Source and destination layers and coefficients alpha and beta appear problem dependent, although small example sets often suffice for selection. Settings transfer from perplexity evaluations to downstream tasks and from MMLU to other single-token-response tasks.Source: §5.2 · pp. 16–17
• Cross-family generalization and normalization requirements remain unresolved, although source-activation normalization improves robustness and one scheme works across Gemma3 model sizes.Other families show qualitatively similar patterns, but Gemma’s Peri-LN architecture and training optimization may yield larger benefits. Further experiments must establish whether normalization is necessary and how to normalize other model families.Source: §5.2 · p. 17
• Blockwise recirculation could alleviate sequential prefill costs by processing token groups in parallel, but its performance trade-off remains unexplored.The proposal processes K tokens together and recirculates them alongside the next K tokens; sequential prefill may otherwise be infeasible for large-context foundation models.Source: §5.2 · p. 17
• Additional recirculation iterations and simultaneous paths remain unexplored extensions that could exploit inexpensive parallel stacks and potentially complementary state representations.The evaluated method recirculates each stack once through a single source–destination pair; proposed extensions would recirculate each stack r times or use multiple pairs. Disjoint effective regions in Gemma3 4B and 12B sweeps motivate the hypothesis that different paths convey state information at different abstraction levels.Source: §5.2 · p. 17
• Adaptive recirculation improves performance with a small tuning set and unchanged model weights, leaving token-conditioned pathways and normalization as further adaptation opportunities.The authors suggest that exploiting internal dynamics could strengthen contextual understanding and support extended, multi-turn reasoning.Source: §5.2 · p. 17

• k + 1 stacks must run for each input step with k recirculation iterations.Source: A Unrolled recirculation architecture · p. 24
• Figure A.1 depicts the two-iteration recirculation architecture as an unrolled transformer.Source: A Unrolled recirculation architecture · p. 24, Figure A.1 · p. 24
• Three passes through each stack result from two recirculation iterations in Figure A.1.Source: Figure A.1 · p. 24

B.1 Hyperparameter sweeps

• 1024-token context windows from training documents were used for the hyperparameter sweeps in Figures 5 and 6.Source: §B.1 · p. 24
• At most two windows were sampled from each document.Source: §B.1 · p. 24
• Windows containing filler tokens were excluded by requiring each document to extend at least to the end of its sampled window.Source: §B.1 · p. 24
• 484 windows from arXiv yielded 495132 predicted tokens.Source: §B.1 · p. 24
• 488 windows from C4 yielded 499224 tokens.Source: §B.1 · p. 24
• 500 windows from PG19 yielded 511000 tokens.Source: §B.1 · p. 24

B.2 Perplexity evaluation

• 10 datasets underpin perplexity evaluation: nine complete evaluation sets and the first 10000 documents from big patent.The complete sets were arXiv, billsum, booksum/books, C4/webtextlike, gov report, lambada, newsroom, PG19, and pubmed. C4 used its validation split; all others used test splits.Source: §B.2 · p. 24
• 1024-token chunks were used for evaluation, excluding partially filled windows except for C4/webtextlike, lambada, and newsroom, whose documents were typically too short.The resulting evaluated token counts are reported in Table 1.Source: §B.2 · p. 24
• Source–destination pairs selected through hyperparameter sweeps for the 1B, 4B, and 12B models were reused throughout the article unless otherwise stated.Table B.1 identifies the selected hyperparameters.Source: §B.2 · p. 24, Table B.1 · p. 25

B.3 Normalization and ramping

• Source-vector scaling helps prevent recirculated activations from overwhelming destination activations as embedding norms increase across transformer layers (Liu et al., 2020).Normalization sweeps measured percentage changes in perplexity on the arXiv, C4, and PG-19 tuning datasets.Source: §B.3 · p. 25
• L2 normalization produces fewer poorly performing hyperparameter settings than no normalization across the evaluated Gemma3 models.Other candidate schemes either showed pathological heatmaps or behaved similarly to simple norm-ratio schemes.Source: §B.3 · p. 25, Figure B.1 · p. 25, Figure B.2 · p. 26
• Convex source–destination mixtures outperform nonconvex mixtures for Gemma3 1B, whereas nonconvex mixtures perform better for Gemma3 4B and 12B.The authors recommend evaluating both mixtures with norm-ratio adjustment of the source. For Gemma3 1B, ramping α_t = min(t/10, 1)α over the first 10 steps slightly further reduces perplexity.Source: §B.3 · p. 25, Figure B.1 · p. 25, Figure B.2 · p. 26

C.1 Robustness across architectures

• The authors hypothesize that Gemma’s compatibility reflects Peri-LN output normalization (Kim et al., 2025) and the optimization process used during training.Unlike input-only normalization, output normalization might prevent diminishing contributions from later layers (Sun et al., 2026). Cross-family sweeps compared Gemma3 1B with Ministral3, Qwen3, Pythia, and Phi2 on arXiv training data.Source: §C.1 · pp. 26–27
• Recirculation gains on Gemma2 and Gemma4 are comparable to Gemma3, although some hyperparameter regions are harmful.These comparisons use arXiv training data. Cross-layer KV-cache sharing and per-layer embeddings may explain instability in smaller Gemma4 variants (Team et al., 2026).Source: §C.1 · pp. 26–27, Figure C.1 · p. 28

C.2 Recirculation versus temperature tuning

• About 1.2 was the optimal softmax temperature both with and without recirculation for Gemma3 1B on PG-19 evaluation data.Evaluation used fully populated 1024-token chunks; the reported combined optimum was restricted to the temperature range tested without recirculation.Source: §C.2 · p. 27, Figure C.2 · p. 28

C.3 Recirculation versus looping

• 250 arXiv training documents supplied two 1024-token subsequences each for the recirculation-versus-looping comparison.Recirculation used α = 0.07, with β = 0.93 for Gemma3 1B and β = 1.0 for the 4B and 12B models.Source: §C.3 · p. 28

C.4 Which tokens benefit from recirculation?

• Token-selective experiments examined downstream effects of individual-token recirculation and compared part-of-speech-selected recirculation against count-matched random-token recirculation.Gemma3 1B PT individual-token tests used 24,960 arXiv training documents, recirculating positions 0–767 and examining lags 1–256. Part-of-speech-selective tests used 3,120 documents with 1024-token sequences.Source: §C.4 · p. 28, Figure C.3 · p. 29

D.1 Instruction following

• 800 instruction-following trials tested identification of a fruit’s or animal’s position in word pairs formed from twenty fruit names and twenty animal names.Gemma3 1B performed essentially at chance. Task-specific layer sweeps estimated an upper bound, while 4B and 12B evaluations also used perplexity-selected layers with α = 0.07.Source: §D.1 · p. 29

D.2 Contextualization

• Contextualization compared recirculation hyperparameters selected by pretrained perplexity, instruction-tuned perplexity, and gender or polysemy five-distractor accuracy using Lepori et al. (2025).The 1B model was not much above chance except on no-distractor polysemy questions.Source: §D.2 · p. 29

• 1531 MMLU development-set problems supported hyperparameter selection for Gemma3 4B PT, with these problems distinct from evaluation examples.Source: §D.3 · p. 29
• Source layer 16, destination layer 5, and α = 0.09 were used across the single-token response datasets.Source: §D.3 · p. 30
• Tuning first swept source–destination pairs at α = 0.07, then scanned α with the pair yielding the lowest perplexity in that sweep.Source: §D.3 · pp. 29–30

GSM8k

• Reported GSM8k results use only the best-performing conditional α, β scheme.The selected scheme is highlighted in Figure 13; this passage supplies no benchmark scores.Source: §D.4 · p. 30
• Training masked the question prompt and optimized Gemma3 4B to predict only the dataset’s ground-truth response.Source: §D.4 · p. 30
• Gemma3 4B used the same hyperparameters as the perplexity experiments in Section 4.6, except for prompt masking.Source: §D.4 · p. 30

D.5 Adaptive recirculation

• 250 documents from each of PG19, C4, and arXiv supply full 1024-token windows for backpropagation-through-time training, followed by evaluation across nine datasets.Perplexity studies use Gemma3 1B PT with previously selected source and destination layers. Evaluation covers ArXiv, PubMed, PG19, BookSum, Lambada, Gov Report, BillSum, OpenWebText, and Big Patent, with nonoverlapping training/evaluation subsets for ArXiv and PG19.Source: §D.5 · pp. 30–32
• A two-hidden-layer GELU MLP predicts token-specific vector coefficients from concatenated source and destination embeddings, following Teoh et al. (2025).Input layer normalization precedes hidden layers matching the model dimension. The conditional-vector formulation outputs twice the model dimension, providing α and β for each dimension.Source: §D.5 · pp. 31–32
• Sigmoid outputs constrain coefficients to [0,1], while initialization sets α = 0.1 and β = 0.9 based on suitable grid-search ranges.Source: §D.5 · p. 32
• Training uses 100 steps, batch size 32, and AdamW, generally freezing Gemma3 parameters.The standard learning rate is 3e-4 with weight decay 1e-4. LLM fine-tuning instead uses learning rate 1e-5 without weight decay; unconditional schemes use 1e-1 without weight decay.Source: §D.5 · p. 32
• Gemma3 4B downstream training masks prompts and trains only on responses, retaining perplexity-experiment hyperparameters and using multiple epochs for the smaller ARC datasets.Single-token benchmarks train on one response token. ARC training processes 3200 examples over 100 steps with batch size 32.Source: §D.5 · pp. 32–33
• Threefold-or-greater improvements in percentage perplexity reduction occur on many datasets with adaptive Gemma3 1B recirculation, with improvements over fixed coefficients on every dataset.The fixed-coefficient comparator uses α = 0.15 and β = 0.85; adaptive recirculation predicts vector-valued α and β for each token.Source: Figure D.6 · p. 32, §D.5 · p. 32