Rotate the Loop: Blockwise Conformality for Looped Transformers

1 INTRODUCTION

• Fixed-depth transformers cannot, under standard complexity assumptions, recognize regular languages or decide graph connectivity, formal limits related to state tracking and multi-step reasoning.These limitations motivate studying depth separately from width or a short chain of thought.Source: §1 · p. 1
• A loop must contract to remain stable while preserving signals long enough for later steps to read them.With weight tying, the same transition map is repeatedly applied, so a lost partial answer may be unavailable when needed.Source: §1 · p. 1
• A free diagonal rescales coordinates independently, letting one direction dominate and disrupting relative angles; blockwise conformality instead gives groups their own rates and rotations.Within the paper’s derivation, a nonnegative real spectrum cannot express the studied state-tracking problems.Source: §1 · p. 2
• The paper introduces a blockwise conformal parameterization to enforce angle preservation and support stable, readable recurrence at test time.Related rotation ideas appear on the token axis and in models with multiple timescales or complex-plane recurrence.Source: §1 · pp. 1–2
• Blockwise conformal transitions improve last-token accuracy on state-tracking problems and language-model quality, including lower validation perplexity and higher LAMBADA.The method constrains the transition geometry while keeping hidden states readable across recurrent steps.Source: §1 · p. 2

Looped and recurrent-depth transformers

• Looped, weight-tied transformers offer variable recurrent depth at test time and have been studied for reasoning, length generalization, and language-model quality.Prior work also examines adaptive trip counts and depth allocated per token.Source: §2 · pp. 2–3

Parcae

• Prior stabilizers constrain transition magnitude or target equilibrium, while Parcae uses a nonnegative diagonal recurrence for stable long trip counts.The cited choice of a simple, cheap diagonal injection motivates exploring a different transition structure.Source: §2 · p. 3

Test-time generalization across recurrent steps

• Earlier test-time generalization methods add recurrence-index information, whereas rotary and related sequence-axis methods apply rotations along token positions.The paper instead places the relevant transition on the recurrent-depth axis.Source: §2 · p. 3

Structured orthogonality and linear recurrences

• Structured orthogonality and complex diagonal state-space models use rotations or magnitude-phase parameterizations; prior work also reports benefits from neighborhoods of orthogonal maps and multiple timescales.RMSNorm is invariant to orthogonal basis changes, although optimizer normalizers can create a privileged coordinate frame.Source: §2 · p. 3

3 MOTIVATION OF READABILITY

• The nonlinear block uses inner products for attention logits, linear readouts, and pre-norm rescaling, so the transition determines whether those comparisons remain valid across steps.The transition operator is the linear map between applications of the same weight-tied block.Source: §3 · p. 3

3.1 A READABLE OPERATOR IS A CONFORMAL ONE

• An operator is readable when it preserves every pairwise inner product up to one shared scale; this is equivalent to the operator being cQ with Q⊤Q = I.A direction-dependent diagonal fails this condition, while pre-norm removes the shared scale from the state passed to the block.Source: §3.1 · pp. 3–4

3.2 WEIGHT TYING IS A CURSE FOR READABILITY

• Repeated application of a non-conformal operator with a strictly dominant largest eigenvalue drives normalized writes toward its dominant eigenspace, whereas conformal transitions preserve readability across depth.For a conformal map, the normalized Gram matrix depends on recurrent steps only through their separation, giving constant diagonals.Source: §3.2 · p. 4

3.3 GROUPS, AND THE FAMILY THEY GENERATE

• Blockwise conformality assigns each coordinate-aligned group its own scale and rotation, preserving readability within each group while allowing distinct timescales.The family ranges from one shared magnitude at G=1 to free magnitudes per plane at G=d/2; Figure 2 depicts the resulting depth-dependent geometry.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

4 BLOCKWISE CONFORMAL INJECTIONS

• The injection partitions coordinates into equal-width groups, pairs coordinates into planes, and assigns each group rates and each plane a frequency.The table evaluates state tracking on A5 and S5, with T=16 and T=32 in-budget and T=48 beyond the training recurrence budget.Source: §4 · p. 5, Table 1 · p. 6
• The transition couples each plane’s rotation angle to its frequency and group rate, so decay and phase share a timescale within each group.Setting all frequencies to zero recovers the earlier nonnegative diagonal transition.Source: §4 · pp. 5–6
• The transition is implemented without materializing its matrix, using the rotary-position-embedding kernel on the recurrent state.The injection module is the only architectural change; the block, data, objective, optimizer, schedule, and other components stay fixed.Source: §4 · p. 6

5.1 BLOCKWISE CONFORMALITY IMPROVES STATE TRACKING

• At T=32, blockwise conformal reaches 0.894 versus 0.692 last-token accuracy on A5; on S5 it reaches 0.808 versus 0.531 at the trained budget.On A5, the advantage remains at T=48, a recurrence budget the checkpoints never saw during training.Source: §5.1 · pp. 6–7
• G=2 peaks in the group-count sweep; G=4 and G=8 fall below the diagonal baseline, while G=256 ends near G=1.The authors attribute the sweep’s shape to phases helping throughout while tied magnitudes at small G provide an additional benefit.Source: §5.1 · pp. 6–7

5.2 A NEIGHBOURHOOD OF CONFORMALITY ACCOUNTS FOR THE GAIN

• At G=2 and T=32, tying rates without rotation reaches 0.810 versus 0.692 for the free diagonal, while combining both choices recovers the full 0.202 gap.Across T=16, T=32, and T=48, neither rate tying nor rotation alone reproduces the full gap.Source: §5.2 · pp. 7–8, Table 2 · p. 7
• At matched spectrum, the pairwise chart reaches 0.721, Householder QR 0.552, the diagonal 0.408, and the Cayley parameterization 0.274.All three orthogonal parameterizations reach Q^TQ = I; the comparison isolates which orthogonal matrices each can reach.Source: Table 3 · p. 7, §5.2 · p. 8

5.3 BLOCKWISE CONFORMALITY IMPROVES LANGUAGE MODELLING

• From T=8 through T=48, the conformal transition is ahead by about 15% on perplexity and 20% on LAMBADA, ending about 17% and 20% ahead, respectively.At T=1, perplexity favors the conformal transition by a few percent, while LAMBADA is a wash or slightly behind at two sizes.Source: §5.3 · pp. 8–9

5.4 THE TRAINED MAP HAS THE PREDICTED GEOMETRY

• The conformal method’s Toeplitz defect stays within precision error across widths and depths, while the diagonal defect grows with recurrent steps and width.The diagonal defect rises from 0.06 at 60M and T=8 toward 0.3 by T=32, alongside its perplexity ceasing to improve with additional steps.Source: §5.4 · p. 9, Table 5 · p. 9, §5.3 · p. 9

6 CONCLUSION

• The blockwise conformal map lets recurrent groups use distinct rates while shrinking together and preserving angles between states.The conclusion frames this geometry as keeping early writes readable at later recurrent steps.Source: §6 · p. 9
• The map addresses state tracking across the depth axis by making the hidden state remain readable across repeated applications of a shared block.The conclusion contrasts readable recurrence with a free map whose repeated application lets one direction dominate and comparisons collapse.Source: §6 · p. 9
• The reported gains cover finite-group word problems and language modelling; the proposed next direction is larger models run for more recurrent steps than during training.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

• The transition-operator construction is a self-contained module replacement, with nothing else changed between comparison methods.Source: REPRODUCIBILITY STATEMENT · p. 10
• Every group-word table uses ten seeds, and every seed is retained.Source: REPRODUCIBILITY STATEMENT · p. 10
• The work uses publicly available data and a public codebase, with configurations and protocols in Sections D and E and code and configurations slated for release.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The study concerns the linear map between passes of a weight-tied transformer.Source: ETHICS STATEMENT · p. 10
• Experiments use publicly available corpora and synthetic word problems over finite groups generated by the authors.Source: ETHICS STATEMENT · p. 10
• The work involves no human subjects, personally identifying information, or new data collection.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

• The authors used a large language model to improve exposition clarity and grammar and to produce plotting and partly implementation scripts.They report reviewing all AI-assisted text and code and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 THE DESIGN SPACE IN ONE PICTURE

• Figure 3 contrasts a shared radius, one radius per group, and free radii, while Table 1 and Table 8 sweeps stop near the family’s shared-radius end.Moving from the baseline frees phase; moving from G=1 toward G=d/2 frees magnitudes group by group.Source: §A.1 · p. 13, Figure 3 · p. 13

A.2 WHAT THE THEORY DOES NOT DECIDE

• The theory does not determine coordinate alignment: the authors treat the basis choice as empirical and compare conformal-factor writings at matched spectra.They report that the restricted L=1 unitary parameterization wins on this task, while the least restricted tested writing finishes below baseline.Source: §A.2 · p. 13

A.3 WHAT CONFORMALITY TRADES AWAY

• Both free diagonal and conformal operators with contraction factors below one converge in norm, but only the non-conformal operator also converges in direction onto its dominant eigenspace.Source: §A.3 · p. 13

A.4 ON IMPORTING SPECTRAL OBSTRUCTIONS

• The imported spectral results rule out a nonnegative real spectrum for arbitrary-length PARITY and motivate freeing phase, but do not establish sufficiency for a fixed operator iterated T times.The authors describe their A5 and S5 result as empirical; input-dependent transitions would fall within the cited sufficiency results.Source: §A.4 · p. 14

A.5 NORMALIZATION

• Pre-normalization makes Corollary 3.3 operational because RMSNorm is invariant to orthogonal basis changes and a global normalizer commutes with Q.A global normalizer cannot undo per-coordinate angle changes, which are the part constrained in Section 3.Source: §A.5 · p. 14

• For a unit probe u and trip count T, the Toeplitz defect is the root-mean-square distance from its Gram matrix to the nearest matrix constant along diagonals.Source: B THE TOEPLITZ DEFECT · p. 14
• In Section 5.4, the defect minimum is averaged over 256 probes drawn uniformly on the sphere.Source: B THE TOEPLITZ DEFECT · p. 14
• For normal A-bar, G_u is Toeplitz for every u if and only if A-bar is conformal.Source: B THE TOEPLITZ DEFECT · p. 14
• For normal invertible A-bar, the defect vanishes for conformal transitions; a nonzero value certifies that the operator is not conformal.Source: B THE TOEPLITZ DEFECT · p. 14

C PROOFS

• Readability is equivalent to M = cQ with Q orthogonal; such a map preserves inner products up to a shared scale, and preserves angles and distances.Source: C PROOFS · p. 14
• For a conformal transition, powers remain conformal and the Gram-matrix comparison depends on the lag between recurrent steps, not their positions.Source: C PROOFS · p. 15
• When a normal operator has unequal eigenvalue magnitudes, a probe exists whose Gram matrix is not Toeplitz; iteration also suppresses non-dominant eigenspace components geometrically.Source: C PROOFS · p. 15
• The blockwise result applies conformal readability within each invariant coordinate group, while distinct groups remain orthogonal under the transition.Source: C PROOFS · p. 15
• At one group, all eigenvalue magnitudes coincide; when each plane is its own group, the magnitudes are unconstrained.Source: C PROOFS · p. 15

D IMPLEMENTATION

• At width 512 with one group, the transition uses 258 scalars against 1024 for the published diagonal; at the 60M width with two groups, it uses 324 against 1280.Source: D IMPLEMENTATION · p. 16

E.1 COMPUTE RESOURCES

• The project used approximately 4000 H100 GPU-hours, with ten seeds for every cell in the group-word study tables.A single group-word run at Tmax=32 with the length curriculum took roughly 2.5 GPU-hours; language-model evaluations and evaluation sweeps reused checkpoints.Source: §E.1 · p. 16

E.2 GROUP WORD PROBLEMS

• Group-word experiments emit every prefix product, use sampled sequences and a length curriculum, and vary only the injection module within each table comparison.Mean token accuracy averages across prefixes, whereas last-token accuracy scores only the full product; T=48 evaluates checkpoints trained with Tmax=32 without retraining.Source: §E.2 · p. 17
• The grouped-diagonal control repeats G learned rates across coordinate-aligned blocks but applies no rotation.Its initialization, optimizer, and schedule match the baseline.Source: §E.2 · p. 17
• The three chart parameterizations produce orthogonal factors but differ in reachable matrices; the spectrum is matched across the four Table 3 rows.The pairwise chart uses fixed coordinate-plane rotations, the Cayley map covers SO(d) up to a null set, and Householder reaches both components of O(d).Source: §E.2 · p. 17

E.3 LANGUAGE MODELS

• The 60M-parameter FineWeb-Edu group-count sweep uses sequence length 2048 and global batch 524k tokens, measures validation perplexity at T=8, and requires G to divide 320 rotation planes.The language-model family evaluates T=1, 8, 16, 32, and 48 by overriding T at load time; T≥16 is beyond the training distribution.Source: §E.3 · p. 17

F PRELIMINARY GROUP COUNT

• 19.08 validation perplexity for G=2 versus 22.45 for the diagonal at full BPTT, selecting two groups for the language-model comparison.The sweep used a 60M model trained at 20 tokens per parameter; full BPTT improved matched pairs without changing their ordering.Source: F PRELIMINARY GROUP COUNT · p. 18, Table 8 · p. 18