PaperlayerView exact revision on arXiv

Source-linked AI summary

Recirculation

Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu

arXiv:2608.17981v1cs.LG

TL;DR

Transformers can struggle to track evolving states, motivating an inference-time recurrence that feeds deep activations back into shallow layers. Recirculation improves perplexity and downstream instruction following, question answering, and problem solving on Gemma3 models with little generation-time overhead but added autoregressive prefill cost.

  • Problem

    Transformers’ parallel operation limits traditional iterative state tracking, contributing to contextualization errors when interpretations evolve.

  • Method

    Recirculation repeatedly runs a transformer while leaking activations from deep layers into shallow layers, with an adaptive variant tuning recurrence coefficients while freezing model weights.

  • Results

    Recirculation improves perplexity and Gemma3 models’ instruction following, question answering, and problem solving across evaluated settings, with robust gains across nine of ten datasets.

  • Takeaways & Limitations

    Recirculation offers a training-free inference-time path for incorporating recurrence into transformers while preserving nearly unchanged generation-time computation.

  • Takeaways & Limitations

    Recirculation’s practical applicability may be limited because optimal hyperparameters can depend on the task, and generalizability beyond Gemma’s architecture requires further investigation.

Abstract

from arXiv · show

We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves remarkable gains on the Gemma3 family, including a 23% reduction in perplexity on a suite of datasets, a 21% increase in accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.

1 State tracking

Transformers’ parallel, depth-bounded processing limits their ability to track evolving belief states, causing contextualization errors and interpretation reversals. Evidence from activation intervention shows that moving a resolved representation from a deep layer to a shallow layer can substantially reduce such errors, motivating recirculation.

  • Motivation: Transformers’ parallel operation during training and prefill precludes traditional iterative state tracking, limiting their ability to maintain evolving belief states.This limitation is linked to contextualization failures, unstable theory-of-mind representations, and the absence of a robust internal world model.
  • Motivation: Without proper state tracking, models can flip-flop between interpretations and fail to recognize inconsistencies, as in the ambiguous meaning of “bank.”A socially adept communicator should acknowledge a meaning reversal after committing to the river-bank interpretation.
  • Evidence for depth-limited state representations: 60% reduction in contextualization errors followed an intervention that copied a deep-layer activation, after ambiguity resolution, into a shallow layer before continuing processing.The experiment patched only a pre-identified critical token in generated questions.
  • Evidence for depth-limited state representations: The intervention suggests that useful state representations may form deep in the transformer but remain insufficiently available to shallower layers during inference.Figure 2a presents transformer depth as a constraint on the utility of state representations, while Figure 2b depicts pushing deep representations toward shallower layers.
  • Motivation for recirculation: Recirculation is motivated as an undifferentiated version of deep-to-shallow activation feedback, but applying it at every token could amplify feedback disastrously without training support.The proposed feasibility of training-free feedback relies on the residual stream functioning as a shared blackboard that aligns representations across layers.

2 Proposed method: Recirculation

Recirculation runs an LLM step-by-step while leaking activations from deep layers to shallow layers, creating recurrence across both transformer depth and input steps. It differs from looped transformers, whose recurrence operates only through repeated shared blocks in depth.

  • Recirculation: Recirculation leaks activation from a deep layer to a shallow layer after each LLM step, with Figure 3a illustrating a possible source–destination pair.The recurrence’s orchestration depends on how the architecture is sequentially unrolled.
  • Recirculation vs. looping: Unlike looped transformers, recirculation propagates state across both depth and input steps, whereas looped-transformer recurrence is solely in depth [Giannou et al., 2023].Recirculation runs two input stacks in parallel at each recurrence step after an initial warm-up.
  • Implementation: The reported experiments use one additional transformer-stack iteration per step, although increasing iterations is possible and unbounded recirculation becomes a true recurrent neural network.The figures depict two executions of the transformer stack at each step; all article experiments use this one-additional-iteration variant.
  • Recirculation: Recirculation mixes activation from source and destination layers, while looping replaces the preceding layer’s input with the entire residual stream.The formalization uses source and destination indices, mixture coefficients α and β, renormalization f(.), and residual-stream states z_i,j,l.

3 Related research

Related work spans looping, recurrent and thinking transformers, stateful training objectives, state tracking, and activation steering. Recirculation is closest to inference-time looping methods but differs conceptually by introducing recurrence rather than simply increasing depth.

  • Looped transformers: Inference-time recurrence methods most closely resemble recirculation, but looping increases effective depth whereas recirculation introduces a distinct recurrence mechanism.Looping can repeat single layers or layer ranges without adding free parameters and may be deterministic or adaptive.
  • Training objectives: Training-based approaches seek stateful transformer embeddings through objectives that steer layers toward exact or approximate state-updating functions.These objectives include losses proposed by Hu et al. (2025), Teoh et al. (2025), and Huang et al. (2026).
  • State tracking: Modern parallel architectures are limited on inherently sequential tasks such as state tracking, multihop inference, and planning because serial capacity is bounded by model depth.Liu et al. (2026) identify this weakness, while cited theory links transformers’ serial capacity to depth.
  • Recurrent transformers: Recurrent transformers use sequential updates that can express arbitrary state dynamics, with implementations operating either token by token or blockwise.State-space models are another related recurrent architecture family.
  • Thinking models: Chain-of-thought and latent-space thinking propagate state through sequential self-signals, enhancing expressivity but restricting parallelism during training.Thinking may use natural-language tokens or latent representations, and training such models can be costly.
  • Activation steering: Activation steering shows that intervening on latent representations can predictably modulate behavior, framing recirculation as inference-time self-steering.Unlike static externally derived steering vectors, the supplied passage contrasts recirculation with modifying the residual stream using such vectors.

4 Experiments

Across model scales, datasets, model families, and downstream tasks, recirculation generally improves perplexity and accuracy, with effects depending on architecture, token position, content, and hyperparameters. Adaptive tuning further improves performance while preserving minimal changes to the original model.

  • Mechanism and token effects: Recirculation’s effects depend on token position and content: middle-architecture source-destination pairs, adverbs, adjectives, verbs, and plural nouns show stronger benefits.The complementary all-but-one-token experiment suggests individual-token benefits are additive in log likelihood, supporting persistent state construction.
  • Perplexity: Up to 16% and 35% perplexity reductions occur for the 1B/4B and 12B models, respectively, with robust gains on nine of ten datasets.LAMBADA is an anomaly because of very short sequences and tokenization artifacts.
  • Cross-model generalization: Recirculation benefits four additional model families, but Gemma3 shows about 5% perplexity reduction versus less than 0.5% for the others.The second and fourth Gemma generations show gains as pronounced as Gemma3, while normalization and α were not explored for the other families.
  • Controls and comparisons: Combining recirculation with temperature tuning yields a 19.55% perplexity reduction, with nearly additive effects that rule out temperature adjustment as the sole explanation.Training-free looping does not produce robust benefits for the Gemma3 family, unlike recirculation.
  • Downstream tasks: Recirculation improves accuracy on six of eight datasets, while also improving both pass@1 and pass@128 for Gemma3 4B.Downstream gains range from modest to significant, and extended generative responses appear to benefit more than single-token responses.
  • Adaptive recirculation: Learning token-conditional vector α and β coefficients performs best among tested adaptive methods, while early-token recirculation can harm Gemma3 1B but not 4B or 12B.The adaptive experiments deliberately tune only recirculation hyperparameters to move minimally beyond the training-free setting.

5 Discussion

The discussion presents recirculation as a training-free, inference-time architectural modification that improves state tracking and downstream capabilities while adding little generation-time computation. It frames the method as a model-guided exploration of architectural affordances, while noting unresolved questions about hyperparameter universality and generalization beyond Gemma.

  • Contribution: Recirculation improves Gemma3 models’ perplexity, instruction following, question answering, and problem solving with almost no additional computation during generation.The method is a training-free, inference-time modification that incorporates recurrence into a transformer for improved state tracking.
  • Relation to prior work: Recirculation is distinct from training-free looping, despite both exploiting alignment in the transformer’s residual pathway.The surrounding architectural literature primarily concerns models trained from scratch or modified during training.
  • Methodological perspective: The work treats recirculation as a methodological contribution in which the model informs recurrence placement, activation mixing, and normalization choices.This contrasts with forcing costly architectural overhauls or relying on arbitrary design choices.
  • Model-design affordances: Recirculation explores model-design affordances: intrinsic foundation-model properties that can be amplified to improve basic operation and capabilities built upon it.The discussion analogizes these affordances to object properties that indicate how an object should be used.
  • Limitations: Recirculation’s practical applicability is limited by task- or domain-dependent hyperparameters and uncertain generalization beyond the Gemma family.The relevant hyperparameters include source and destination layers and recirculation coefficients α and β; Gemma’s Peri-LN architecture and training optimization may yield unusually large benefits.

A Unrolled recirculation architecture

The unrolled recirculation architecture executes multiple passes through each transformer stack. With two recirculation iterations, it uses three passes; generally, k iterations require k + 1 stacks per input step.

  • A Unrolled recirculation architecture: For k recirculation iterations, each input step requires running k + 1 stacks.
  • A Unrolled recirculation architecture: Two recirculation iterations yield three passes through each transformer stack in the unrolled architecture.Figure A.1 depicts this configuration.

B Recirculation implementation details · B.1 Hyperparameter sweeps

Hyperparameter sweeps used 1024-token contexts drawn from roughly 500 filler-free windows across arXiv, C4, and PG19 training documents.

  • B.1 Hyperparameter sweeps: Hyperparameter sweeps used a 1024-token context window and roughly 500 windows from arXiv, C4, and PG19 training documents.At most two windows per document were sampled, and each window had to extend to the document’s end without filler tokens.
  • B.1 Hyperparameter sweeps: The filtering procedure yielded 484 arXiv windows, 488 C4 windows, and 500 PG19 windows.These corresponded to 495132, 499224, and 511000 predicted tokens, respectively.

B.2 Perplexity evaluation · B.3 Normalization and ramping

The perplexity evaluation spans ten datasets using fixed-length document chunks and tuned source–destination hyperparameters. Normalization sweeps compare recirculation schemes, finding L2-based methods more robust than no normalization, while ramping modestly improves Gemma3 1B.

  • B.2 Perplexity evaluation: Perplexity was evaluated across ten datasets, using complete evaluation sets for nine and the first 10,000 big-patent documents.The evaluation split was validation for C4 and test for the other datasets.
  • B.2 Perplexity evaluation: Documents were divided into 1,024-token chunks, excluding partially filled windows except for C4/webtextlike, lambada, and newsroom.
  • B.2 Perplexity evaluation: The reported results use source–destination hyperparameters selected through sweeps for the 1B, 4B, and 12B models.These settings were used throughout the article unless otherwise noted, as summarized in Table B.1.
  • B.3 Normalization and ramping: Normalization sweeps test source-vector scaling because embedding norms increase across transformer layers, potentially allowing source activations to overwhelm destination activations.Figures B.1–B.3 report perplexity percentage changes across the arXiv, C4, and P tuning datasets.
  • B.3 Normalization and ramping: The normalization schemes and their notation are summarized in Table B.2, which rewrites embeddings by unrolling step, input step, and layer.
  • B.3 Normalization and ramping: L2 normalization schemes are better behaved than no normalization, producing fewer hyperparameter settings with poorer performance.The no-normalization scheme still attains reasonable outcomes, while a convex combination of source and destination vectors is reported as superior in the supplied passage.
  • B.3 Normalization and ramping: For Gemma3 1B, ramping the recirculation coefficient over the first 10 steps produced a small additional perplexity reduction.The coefficient was defined as α_t = min(t/10, 1)α for t ≥ 0.

C Basic results … D.2 Contextualization

The supplementary evaluations test recirculation across architectures, temperature tuning, looping, token types, instruction following, and contextualization. Together, they characterize where recirculation transfers, how it is configured, and how downstream evaluations are constructed.

  • C.1 Robustness across architectures: Across Ministral3, Qwen3, Pythia, and Phi2, Figure 7 compares recirculation hyperparameter sweeps using the arXiv training set and Gemma3 1B PT settings.The shared settings scale the source layer norm to the target layer norm, with α = .07 and β = 1 −α.
  • C.1 Robustness across architectures: Recirculation gains on older and newer Gemma generations are comparable to Gemma3, although some Gemma4 hyperparameter regions are harmful.Figure C.1 uses α = 0.07; β = 1.0 for Gemma4 and β = 1 −α for Gemma2.
  • C.2 Recirculation versus temperature tuning: Temperature tuning is evaluated against recirculation on Gemma3 1B using PG-19 1024-token chunks, with Figure C.2 separating temperature-only and combined sweeps.The experiments sweep softmax temperatures broadly enough to identify where adjustment improves perplexity; partially filled sequences are excluded.
  • C.3 Recirculation versus looping: The looping comparison evaluates recirculation on 250 arXiv training documents, using two 1024-token subsequences per document and model-size-specific β settings.Recirculation uses α = 0.07, β = 0.93 for the 1B model, and β = 1.0 for the 4B and 12B models.
  • C.4 Which tokens benefit from recirculation?: Token-level analyses examine downstream effects after individually recirculating tokens 0–767 across 24,960 arXiv documents and lags 1–256.The setup uses Gemma3 1B PT, 1024-token contexts, and randomly selected document positions.
  • C.4 Which tokens benefit from recirculation?: Part-of-speech experiments compare recirculating all tokens tagged with a given part of speech against count-matched random tokens.Figure C.3 uses 3,120 arXiv documents, each contributing a 1024-token sequence from a random document position.
  • D.1 Instruction following: Instruction-following evaluations comprise 800 fruit-versus-animal position-identification trials, split evenly between responding to fruits and responding to animals.Responses are selected from eight candidate tokens representing first and second with capitalization and spacing variants.
  • D.2 Contextualization: Contextualization experiments select hyperparameters from pretrained perplexity minimization and additionally sweep the Gemma3 instruction-tuned models on gender and polysemy conditions.The task-specific sweeps appear in the upper-right portions of Figures D.2–D.4 for the 1B, 4B, and 12B models.

D.3 Multiple-choice and single-token response tasks

The study tested Gemma3 4B PT on multiple-choice and single-token response tasks, selecting recirculation hyperparameters through sweeps on 1531 MMLU development-set problems distinct from evaluation examples. The resulting settings were source layer 16, destination layer 5, and α = 0.09.

  • D.3 Multiple-choice and single-token response tasks: 1531 MMLU development-set problems were used to sweep source and destination layers for Gemma3 4B PT, with α fixed at 0.07.The problems were distinct from the examples used for evaluation.
  • D.3 Multiple-choice and single-token response tasks: The final recirculation settings for single-token response datasets were source layer 16, destination layer 5, and α = 0.09.These values were selected after scanning α while fixing source and destination to the lowest-perplexity pair from the initial sweep.
  • D.3 Multiple-choice and single-token response tasks: A separate α scan fixed the source and destination layers to the pair yielding the lowest perplexity in the source–destination sweep.The scan provided the hyperparameters used across the single-token response datasets.

D.4 Standard benchmark datasets: GSM8k

On GSM8k, the Gemma3 4B model was trained with the perplexity-experiment hyperparameters, masking the question prompt and predicting only the dataset’s ground-truth response.

  • D.4 Standard benchmark datasets: GSM8k: Gemma3 4B was trained with the perplexity-experiment hyperparameters, masking the question prompt and using only the ground-truth response as the prediction target; results use the best-performing conditional α, β scheme.The reported scheme is highlighted in Figure 13.

D.5 Adaptive recirculation

Adaptive recirculation uses a learned, token-dependent MLP to produce vector-valued α and β coefficients, outperforming fixed-coefficient recirculation in perplexity reduction across every evaluated dataset. The method freezes Gemma3 parameters in most simulations and is trained with lightweight optimization settings, with downstream evaluation adapting the training objective to response tokens.

  • Method: The adaptive scheme uses a two-hidden-layer GELU MLP that receives concatenated source and destination embeddings and outputs coefficient values constrained to [0,1].The MLP uses the model hidden size, layer normalization at its input, and sigmoid output activation; parameters are initialized with α = 0.1 and β = 0.9.
  • Training setup: Training runs for 100 steps with batch size 32 using AdamW, while Gemma3 parameters remain frozen for simulations other than LLM fine-tuning.The default learning rate is 3e-4 with weight decay 1e-4; LLM fine-tuning instead uses learning rate 1e-5 without weight decay.
  • Evaluation: Perplexity evaluation covers nine datasets, with nonoverlapping ArXiv and PG19 subsets reserved between training and evaluation.The evaluated datasets are ArXiv, PubMed, PG19, BookSum, Lambada, Gov Report, BillSum, OpenWebText, and Big Patent.
  • Results: Adaptive recirculation increases perplexity reduction on every dataset versus fixed coefficients, by a factor of three or more on many datasets.The comparison is between fixed α = 0.15, β = 0.85 and an MLP producing vector-valued α and β for each token.
  • Downstream training: For downstream Gemma3 4B tasks, the same hyperparameters are reused while masking the prompt and training only on the response token.ARC datasets receive multi-epoch training because they contain fewer examples than MMLU.
Loading 2608.17981v1…