PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Looped Transformer

Yifan Zhang, Jichen Feng, Shihan Qin

arXiv:2610.07591v1cs.CLcs.AIcs.LG

TL;DR

State tracking requires updating a state with each input, but a Transformer applies a fixed number of layers at each position. The Recurrent Looped Transformer pairs a parallel causal encoder with a recurrent decoder that feeds back its previous final state. Across algorithmic tasks, it generalizes beyond training lengths where an eight-layer Transformer remains at chance on parity and swap-based S5, and its best splits outperform the Transformer on modular arithmetic.

  • Problem

    State tracking requires updating a state with each input, while a Transformer computes each position with a fixed number of layers.

    Source: §1 · p. 3

  • Method

    RLT divides its layers between a parallel causal encoder and a recurrent decoder that merges each encoder representation with the previous token’s final decoder output.

    Source: §1 · p. 3

  • Results

    With eight layers, RLT generalizes parity and swap-based S5 tracking far beyond training lengths where an eight-layer Transformer is at chance, and its best splits outperform the Transformer on modular arithmetic beyond training lengths.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The best encoder–decoder depth split depends on the task, and the feedback interval trades known-token parallelism against accuracy, especially for permutation tracking.

    Source: §1 · p. 3, §2.7 · p. 7

    The sequence formulation assumes an independent sequence beginning with BOS, with the same conditional distribution before and after the prompt boundary.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

State tracking requires an update at every input, but the depth a Transformer applies to each token is fixed regardless of sequence length. We introduce the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder and a recurrent decoder. At each token, the decoder merges the encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost. On six algorithmic tasks, we compare five splits of eight layers with an eight-layer Transformer over three seeds. Trained on at most 40 bits, two RLT splits generalize parity to 256 bits with 100% accuracy in every seed, while the Transformer stays at chance. On swap-based $S_5$ permutation tracking at eight times the training length, RLT reaches 97% final-state accuracy versus under 1% for the Transformer, and accuracy increases with decoder depth. On modular arithmetic beyond the training lengths, RLT reaches up to 93% versus 33% for the Transformer. Ablations show that these gains depend on the feedback: removing it drops parity and swap-based $S_5$ to chance at every split. Updating the feedback once per four-token chunk lets known tokens in a chunk run in parallel and keeps 64-bit parity at 99%, while permutation tracking depends on per-token feedback: chunking lowers length-64 swap-based $S_5$ from 100% to 20%.

1 Introduction

The Recurrent Looped Transformer (RLT) adds recurrent state updates to a parallel causal encoder, extending the computation path across tokens while keeping per-token depth fixed. Across algorithmic tasks, its length generalization depends on the task, decoder depth, and feedback frequency.

1 Introduction

  • State tracking requires updating a state with each input, while a Transformer applies a fixed number of layers at every position.The passage notes that, unless TC0 = NC1, fixed-depth Transformers cannot track compositions of S5 permutations over arbitrarily long sequences.Source: §1 · p. 3
  • RLT combines a parallel causal encoder with a recurrent decoder that merges each encoder representation with the previous token’s final decoder output before predicting the next token.For decoder depth LD, processing t tokens creates a recurrent path through tLD decoder blocks while each token passes through only LE + LD layers.Source: §1 · p. 3
  • The feedback interval B and encoder–decoder depth split are design choices: larger decoder allocation lengthens the recurrent path but adds token-ordered work, while larger B enables chunked feedback.For T known tokens, the decoder uses TLD sequential block stages at B = 1, ⌈T/B⌉LD at finite B, and LD at B = ∞; RLT-2 shares RLT-1’s parameters.Source: §1 · p. 3
  • The evaluation compares five eight-layer allocations with an eight-layer decoder-only Transformer on six algorithmic tasks, using three seeds and lengths beyond training.Source: §1 · p. 3
  • At 256 bits, RLT splits 5+3 and 7+1 reach 100±0% parity accuracy in every seed, versus 50.07 ± 1.63% for the Transformer; on swap-based S5 at 256 operations, 4 + 4 reaches 97.30 ± 2.76% versus 0.85 ± 0.30%.After 5,000 training steps, 6 + 2 reaches 93.36 ± 5.69% on flat mod-5 expressions of length 63, compared with 33.20 ± 2.33%.Source: §1 · p. 3
  • Parity generalizes best with one or three decoder layers, whereas swap-based S5 accuracy at 256 operations rises with decoder depth, from chance with one decoder layer to 97% with four.Source: §1 · p. 3
  • Without feedback, parity and swap-based S5 stay near chance from length 64 at every split; four-token chunks retain 98.99 ± 1.66% 64-bit parity but lower 64-operation swap-based S5 from 100% to 19.60 ± 6.54%.Source: §1 · p. 3
  • Unlike related feedback-based models, RLT separates recurrence from parallel encoder memory and treats the allocation of depth between encoding and recurrent updates as a design variable.The passage names Feedback Transformer, Recurrent Transformer, Full-bandwidth Transformer, T2MLR, and Latent Recurrent Transformer as related approaches.Source: §1 · p. 4

2 Method

RLT pairs a causal encoder with a recurrent decoder that merges each token’s encoder representation with prior decoder state. Its feedback interval controls how often that state updates, trading a longer recurrent path for more parallel processing of known tokens.

2.1 Sequence and state

  • The recurrent state comprises decoder output and sliding-window attention caches, initialized before BOS and updated for each observed or sampled token.Generation encodes each sampled token incrementally and updates both components of the state.Source: §2.1 · p. 4, §2.3 · pp. 4–5

2.2 Causal encoder and memory

  • Causal masking lets encoder positions run together, while prefix-restricted cross-attention exposes each decoder position only to encoder memory through that position.Memory may be shared across decoder layers or separately projected for each layer.Source: §2.2 · p. 4

2 Method

  • RLT divides its layers between a causal encoder and recurrent decoder, merging each encoder representation with the previous state before predicting the next token.The encoder builds token representations and key–value memory; the decoder attends to the encoder prefix and its recent activations.Source: §2.2 · p. 4, §2.1 · p. 4, §2.3 · p. 5

2.4 Prompt prefill and incremental decoding

  • Prompt prefill computes encoder representations and memory first, then processes decoder positions in order with attention restricted to each position’s prefix.Incremental generation appends encoder memory and updates the decoder state and attention caches for every token.Source: §2.4 · p. 5, §2.3 · p. 5

2.5 Merge and decoder blocks

  • A gated merge combines the encoder output and prior state; each decoder block applies causal sliding-window attention, encoder-memory cross-attention, and an FFN.The main comparison uses feedback scale α = 0.1.Source: §2.5 · p. 5

2.6 Recurrent depth and execution cost

  • After T prompt and response tokens, the recurrent path traverses (T + n)LD decoder blocks, while each token evaluates LE + LD blocks.For LE = LD = 48, Figure 3 depicts 48t decoder blocks along the path after t tokens and 96 blocks evaluated per token.Source: Figure 3 · p. 6, §2.6 · p. 6

2.7 Feedback interval: RLT-1, RLT-2, and RLT-0

  • RLT-1 updates feedback every token, RLT-2 at chunk boundaries, and RLT-0 removes feedback; larger chunks batch known positions and reduce sequential decoder stages.For a known prefix of length T, the decoder needs ⌈T/B⌉LD sequential block stages; causal SWA caches persist between batches, and generation remains token-by-token.Source: §2.7 · pp. 6–7, Figure 5 · p. 9, Known-token decoder batches · p. 9

2.8 Training and state replay

  • Training uses teacher forcing, cross-entropy, and full backpropagation through recurrent outputs, decoder caches, and encoder memory; replay rebuilds state after parameter updates.For reinforcement-learning evaluation after an update, the sampled response history must be replayed under the current parameters.Source: §2.8 · p. 7, Known-token decoder batches · p. 9

3 Algorithmic Experiments

The experiments test how decoder depth and feedback frequency affect length generalization across six algorithmic tasks. They compare five eight-layer RLT allocations with an eight-layer Transformer using three seeds and shared evaluation data.

3 Algorithmic Experiments

  • The study tests how decoder depth and feedback frequency affect length generalization across addition, parity, modular arithmetic, and standard and swaps-based S5 tracking.It first compares encoder–decoder allocations, then varies feedback interval at a fixed split.Source: §3 · p. 9

3.1 Models and evaluation

  • All models have eight logical layers; the five RLT-1 allocations are compared with an eight-layer decoder-only Transformer using shared training and test data and three initialization seeds.The models use width 512, four attention heads, and task-specific training lengths and checkpoint selection based on in-distribution validation loss.Source: §3.1 · pp. 9–10

3.2 Length extrapolation and decoder allocation

  • At 256 bits, RLT 5 + 3 and 7 + 1 reach 100 ± 0% parity accuracy versus 50.07 ± 1.63% for the Transformer; at 256 operations, RLT 4 + 4 reaches 97.30 ± 2.76% swaps-S5 accuracy versus 0.85 ± 0.30%.At flat mod-5 length 63, RLT 6 + 2 reaches 93.36 ± 5.69%, compared with 33.20 ± 2.33% for the Transformer.Source: §3.2 · p. 10
  • After 5,000 steps, RLT 6 + 2 reaches 93.36 ± 5.69% on flat mod-5 expressions of length 63, compared with 33.20 ± 2.33% for the Transformer.Accuracy falls on longer expressions, and several flat mod-5 splits vary widely across seeds.Source: §3.2 · p. 10

3.3 Feedback frequency and parallelism

  • At 64 bits, RLT-1 reaches 100 ± 0% parity accuracy, chunk4 reaches 98.99 ± 1.66%, and RLT-0 reaches 50.23 ± 3.80%; at 64 operations, swaps-S5 reaches 100 ± 0% with RLT-1 and 19.60 ± 6.54% with chunk4.RLT-0 and the Transformer are near the 1/120 uniform reference on swaps-S5 at this length.Source: §3.3 · p. 10
  • Chunk4’s effect depends on the split: at 256 bits, 4 + 4 retains 69.34 ± 24.17% parity accuracy, while 7 + 1 and 8 + 0 are near chance at 64 bits.For swaps-S5, chunk4 5 + 3 reaches 50.16 ± 18.77% at 64 operations, compared with 19.60 ± 6.54% for 4 + 4.Source: §3.3 · pp. 10, 12

3.4 Addition, standard S5, and scope

  • All models reach 100% teacher-forced accuracy on the training-width addition validation set, but eight-layer model means at 32 digits range from 14.89% to 16.84%; standard S5 stays near the 1/120 reference.The experiments measure supervised algorithmic performance; reinforcement-learning performance is not evaluated.Source: §3.4 · p. 12

4 Related Work

Related work explores cross-token feedback, recurrence over blocks or segments, depth sharing, and reusable encoder memory. RLT combines parallel encoder memory with recurrent decoder updates across tokens.

Hybrid Transformer–RNN models

  • RLT-1 combines parallel Transformer encoding with recurrent decoding, unlike hybrid Transformer–LSTM machine-translation models (Chen et al., 2018).Its decoder uses encoder-memory cross-attention and sliding-window attention for causal language modeling.Source: §4 · p. 12

Encoder-derived memory

  • YOCO and DeepSeek-V4.1-Flash reuse encoder-derived or encoder-projected KV memory, while RLT-1 also builds global encoder memory but processes every prompt token with its recurrent decoder (Sun et al., 2024).Source: §4 · p. 12

Temporal feedback

  • Feedback Transformer and Recurrent Transformer use cross-token feedback, while RLT-1 feeds the preceding final decoder output through a gated merge and keeps encoder memory separate (Fan et al., 2020; Oncescu et al., 2026).RLT-1's recurrent dependency spans the full decoder; Recurrent Transformer feedback is layerwise.Source: §4 · pp. 12–13

Block- and segment-level recurrence

  • Block-Recurrent Transformers process token blocks in parallel between recurrent steps, while RLT-2 updates feedback at chunk boundaries without dedicated memory tokens or a separate recurrent-state bank (Hutchins et al., 2022; Bulatov et al., 2022).RLT-1 instead feeds back at every token and requires sequential decoder work within each block.Source: §4 · p. 14

Weight sharing across depth

5 Conclusion

RLT pairs a parallel causal encoder with a decoder that feeds its final state back at every token. Across the tested algorithmic tasks, it generalizes farther than an eight-layer Transformer on parity, swap-based S5, and modular arithmetic, while chunked feedback trades some accuracy for parallelism.

5 Conclusion

  • The decoder feeds its final state back at every token, making the computation path grow with sequence length at a fixed per-token cost.Source: §5 · p. 14
  • RLT generalizes parity and swap-based S5 far beyond training lengths, and its best splits outperform the Transformer on modular arithmetic beyond training lengths.Source: §5 · p. 14
  • Removing feedback eliminates the gains on parity and swap-based S5; four-token chunks retain 99% 64-bit parity while swap-based S5 benefits most from per-token feedback.Source: §5 · p. 14
  • The best encoder–decoder split depends on the task: parity generalizes best with one or three decoder layers, while swap-based S5 improves with each additional decoder layer up to four tested.Source: §5 · p. 14

A Architecture and Execution Details

The execution details distinguish parameter sharing, serving consistency, sequential prefill, batching, and memory reuse. Encoder work can be parallelized across known positions, but recurrent decoder transitions remain sequential within each sequence.

A.1 Sharing encoder and decoder weights

  • With tied weights, encoder self-attention and decoder sliding-window attention share compatible projections and FFNs, while cross-attention, normalization, merge, and readout remain separate.Untied encoder and decoder weights preserve the same recurrence; memory groups may be shared in either version.Source: §A.1 · p. 18

B.1 Prompt–response consistency

  • Under the stated exact-arithmetic and deterministic-operation assumptions, batched encoder prefill followed by recurrent decoder updates yields the same states and next-token distributions as incremental processing.The conditional distribution for a fixed token history is unchanged when the prompt–response split moves.Source: §B.1 · p. 18

B.2 Prefill work and sequential depth

  • Prefill requires T sequential decoder transitions, each containing LD blocks, which can reduce hardware utilization even when arithmetic complexity is comparable to a dense Transformer.Each new token evaluates encoder and decoder blocks plus merge and memory projection; attention work grows with context length.Source: §B.2 · pp. 18–19

C.1 Batching independent sequences

  • For known sequences or prompts, encoder features and memory projections are computed in parallel across positions, then decoder positions are processed in order; independent sequences can batch their next updates.Small batches and uneven sequence lengths may limit utilization.Source: §C.1 · p. 19

C.2 Memory traffic and parameter reuse

  • Fewer encoder-memory groups reduce KV storage and projection work, while more groups permit layer-specific transformations; normalization, gating, state projection, and residual addition can be fused into one kernel.Source: §C.2 · p. 19

C.3 Training memory and numerical agreement

The section describes reducing training memory through recomputation and aligning sampler and trainer behavior for numerical agreement.

C.3 Training memory and numerical agreement

  • Activation checkpointing reduces stored activations by recomputing them during backpropagation through time.The trade-off is recomputation during BPTT.Source: §C.3 · p. 20
  • Sampler and trainer implementations must align positional conventions, stochastic behavior, and cache contents.Numerical agreement should be checked across execution modes.Source: §C.3 · p. 20
  • The executed policy depends on precision, cache construction, reductions, and sampling transforms as well as weights.These are among the execution details to align and check for numerical agreement.Source: §C.3 · p. 20

D Training Objectives

This section specifies RLT training objectives and execution, including recurrent state handling, replay, gradient paths, and chunked feedback. It also describes constraints on cache reuse and numerical agreement.

D.1 Autoregressive pretraining

  • Full-sequence next-token prediction trains the model jointly, with state reset at independent documents and context specified for segmented training.Every non-BOS target contributes to the loss; dropping earlier state changes the history on which likelihood is conditioned.Source: §D.1 · p. 20

D.2 Supervised fine-tuning

  • SFT masks select assistant-target loss terms, but user, system, and tool tokens still update decoder state and carry gradients.The model continues from the existing state when an assistant turn begins.Source: §D.2 · p. 20

D.3 Policy gradients and replay

  • Policy replay rebuilds the sampled history under current parameters, while recorded behavior probabilities remain fixed for importance ratios.Exact importance sampling requires target-policy support to be covered by behavior probabilities; tokenwise clipping can introduce bias.Source: §D.3 · p. 21

E Multi-turn Inference and Cache Reuse

  • Conversations form one token history: user and tool inputs update state, and editing an earlier token invalidates later cached state.Recompute from a valid checkpoint before continuing after an edit or truncation.Source: E Multi-turn Inference and Cache Reuse · p. 21

F Execution Procedures

  • Execution procedures preserve token-order decoder updates, and full-BPTT RL replay retains gradients through prompt and external-token state updates.Prompt replay under no_grad or detached prompt KV drops prompt-state gradients required by full BPTT.Source: §F.1 · p. 23, §F.2 · p. 23, §F.4 · p. 24

G Causality and Gradient Paths

  • Under deterministic execution, each recurrent state depends only on the input prefix through its position.Detaching recurrent outputs or decoder KV removes corresponding gradient terms even when forward probabilities stay unchanged.Source: G Causality and Gradient Paths · pp. 24–25

H When Cached States Can Be Reused

  • A detached cache may preserve correct values but lacks the computation graph needed for full BPTT; SWA eviction does not erase earlier gradient paths.Training may therefore require more activation storage than the inference cache alone.Source: H When Cached States Can Be Reused · pp. 25–26

I RLT-2: Chunk-Parallel Hidden-State Feedback

  • RLT-2 processes known tokens within a chunk together, then uses the last-position output as the next feedback state.Chunking changes forward dependencies, while gradient truncation and optimizer-update frequency are separate choices; execution kernels can also alter logits and boundary states.Source: §I.1 · p. 26, §I.2 · p. 27, §I.3 · p. 27

I.4 Training and inference efficiency

This section analyzes RLT training and inference costs and describes a controlled benchmark for efficiency and execution comparisons.

I.4 Training and inference efficiency

  • RLT-2 retains per-token decoder arithmetic but groups sequential decoder batches into K chunks, permitting larger matrix operations and weight reuse across up to B positions.RLT-1 and RLT-2 both add O(Td^2) merge work.Source: §I.4 · p. 28
  • During generation, RLT-2 can reuse its feedback projection until the next boundary, while decoder blocks and KV writes remain per-token operations.RLT-1 and RLT-2 add O(d) feedback storage; chunk batches also require temporary activations and current-chunk KV during prefill.Source: §I.4 · p. 28
  • A controlled benchmark would sweep B from 1 to 32 and compare RLT-0 at fixed data, model, optimizer, batch, precision, and kernel settings.The proposed measures include accuracy, training throughput, latency, and peak device memory.Source: §I.4 · p. 28

J.1 Complete six-task study at 2,000 steps

  • The appendix reports 108 runs across six architectures, six tasks, and three seeds, with length generalization measured from each run’s best in-distribution checkpoint.Runs were trained for 2,000 optimizer steps.Source: §J.1 · p. 28

J.1.1 Models, tasks, and evaluation protocol

The study evaluates six algorithmic tasks using three seeds and reports validation and length-generalization results. RLT performance varies by task, with strong results on parity and swap-based S5 but substantial variability on flat modular arithmetic.

J.1.2 Parity: learning speed and initialization variability

  • At step 2,000, RLT-1 splits 4 + 4 through 7 + 1 reach 100 ± 0% parity accuracy, while the Transformer reaches 94.84 ± 3.43%.All three RLT-1 6 + 2 seeds reach 100% by step 600; at step 500, 6 + 2 reaches 99.44 ± 0.98% versus 48.48 ± 0.53% for the Transformer.Source: Table 2 · p. 29, §J.1.2 · p. 29

J.1.4 Addition and permutation state tracking

  • At step 2,000, all six architectures reach 100 ± 0% addition accuracy, and RLT-1 splits 4 + 4, 5 + 3, and 6 + 2 reach 100 ± 0% swaps-S5 final-state accuracy.For swaps-S5, 7+1 and 8+0 average 99.35±0.81% and 99.61±0.68%, compared with 99.09±0.23% for the Transformer.Source: Table 2 · p. 29, §J.1.4 · p. 31

J.1.3 Modular arithmetic with and without brackets

  • At 2,000 steps, flat modular arithmetic gives RLT-1 5 + 3 a 94.18 ± 7.29% mean, compared with 64.02 ± 37.64% for the Transformer.Bracketed-expression RLT-1 means range from 70.53% to 75.17%, compared with 73.87±9.07% for the Transformer.Source: §J.1.3 · pp. 29, 31

J.1.4 Addition and permutation state tracking

  • On standard S5, mean final-state accuracy ranges from 0.52–2.47%, against a uniform 120-class reference of 0.83%.Mean prefix-token accuracy is 5.40–7.99% and includes easier early states.Source: Table 2 · p. 29, §J.1.4 · p. 31

J.1.5 Length generalization

Length-generalization tests compare models on shared examples beyond their training ranges, using teacher-forced answer-token accuracy for addition and final-label or final-state accuracy for formal tasks. RLT results vary by task and depth split, with strong swaps-S5 and parity results but weaker long-length arithmetic performance.

Addition and parity

  • At nine digits per operand, addition accuracy is 68.05 ± 3.64% for RLT-1 7 + 1 and 58.61 ± 2.44% for the Transformer; by 32 digits, model means are 14.89%–16.84%.Addition scores teacher-forced answer tokens on 256 pairs per operand width.Source: §J.1.5 · p. 31, Figure 12 · p. 32, Table 3 · p. 34
  • At 256 bits, parity accuracy is 100 ± 0% for splits 5 + 3 and 7 + 1, versus 50.07 ± 1.63% for the Transformer.Other RLT splits range from 66.76 ± 28.78% to 84.05 ± 27.63% at this length.Source: §J.1.5 · p. 31, Figure 12 · p. 32, Table 3 · p. 34

Modular arithmetic

  • At flat mod-5 length 63, RLT-1 5 + 3 reaches 66.96 ± 40.04%, versus 24.48 ± 3.89% for the Transformer, but all model means approach the 20% reference by length 255.Bracketed mod-5 accuracy also declines as expressions grow longer.Source: §J.1.5 · p. 31, Figure 12 · p. 32, Table 3 · p. 34

Permutation state tracking

  • At 256 swaps-S5 operations, RLT-1 4 + 4 reaches 97.30 ± 2.76% final-state accuracy versus 0.85 ± 0.30% for Transformer 8.At 512 operations, 4 + 4 retains 55.70 ± 25.78% final-state accuracy and 91.16 ± 6.09% prefix-token accuracy; standard S5 final-state accuracy remains 0.72%–1.43%.Source: §J.1.5 · p. 31, Table 3 · p. 34
  • S5 is scored by prefix-token, final-state, and whole-sequence accuracy, which can differ because a correct final state may follow incorrect intermediate predictions.Whole-sequence accuracy requires every prefix prediction to be correct.Source: §J.1.5 · p. 31, Figure 13 · p. 33

J.1.6 Training stability and comparison scope

The appendices document training instability, model-size accounting, an RLT-0 control, metric aggregation, and the completed training snapshot. They specify how comparisons and supporting artifacts were produced.

J.1.6 Training stability and comparison scope

  • Seed 42 parity accuracy for RLT-1 5 + 3 falls from 100% at step 500 to 48.05% at step 600, then returns to 100% at step 700.The authors note that final averages can hide unstable learning trajectories.Source: §J.1.6 · p. 34

J.2 Parameter counts

  • Table 4 lists unique trainable parameters; RLT-1 decoder blocks have SWA and encoder-memory cross-attention, while Transformer blocks have one attention sublayer.RLT-1 8 + 0 applies the learned state projection and gated merge at every token.Source: §J.2 · p. 35, Table 4 · p. 35

J.3 Implementation of RLT-0

  • The rlt_parallel_tiny RLT-0 control uses an untied 4 + 4 encoder–decoder split, width 512, and 27,938,944 unique parameters.It follows the paper's RLT architecture and trains with full backpropagation.Source: §J.3 · p. 35

J.4 Seed aggregation and metric definitions

  • Validation curves aggregate all three seeds at the same optimizer step without interpolation or imputation, while task metrics use different scoring units.Parity and mod-5 score one final label; S5 reports final-state or prefix-token accuracy, and addition uses teacher-forced answer-token accuracy.Source: §J.4 · p. 35

J.5 Reproducing the completed training snapshot

  • All 108 runs completed 2,000 steps; the manifest records exact collection timestamps and source hashes, and plot_depth8.py rebuilds figures and tables from 2,160 validation and 216,000 training-step records.Source: §J.5 · pp. 35, 37

J.6 Length-generalization protocol and supplementary metrics

The appendix specifies shared-data length tests, checkpoint selection, supplementary scoring, and feedback-variant experiments. It also reports checkpoint sensitivity, initialization variation, runtime comparisons, and the mixed results of reducing feedback scale.

J.6 Length-generalization protocol and supplementary metrics

  • Checkpoint selection uses minimum ID-validation example loss, and models share test examples across configured lengths and tasks.The evaluation covers 47 shared test sets and 846 model–seed–length evaluations; addition uses 256 pairs, while formal-task sets use 1,024 sequences.Source: §J.6 · p. 38

J.7 Token-accuracy trajectories at lengths 32 and 33

  • Length-32 and length-33 validation plots compare formal-task token accuracy across 90 runs, with S5 scoring prefix states and parity or mod-5 scoring final labels.Flat mod-5 uses length 33 because its expressions have odd payload lengths.Source: Figure 16 · p. 39, §J.7 · p. 42, Figure 18 · p. 42

J.8 Sixteen-layer parity: best and final checkpoints

  • At 256 bits, four sixteen-layer RLT-1 splits retain 100% parity accuracy at both best and final checkpoints, versus 49.41% for Transformer 16.Accuracy is unchanged in 74 of 80 model–length comparisons.Source: §J.8 · p. 43, Table 7 · p. 43, Figure 19 · p. 44

J.10 Mod-5 feedback variants at 5,000 steps

  • At flat mod-5 length 63, chunk4 4 + 4 reaches 50.78 ± 38.02%, versus 44.43 ± 43.81% for RLT-1, while seed-42 chunk4 training steps are 2.27 times as fast.On bracketed mod-5 at length 64, RLT-1 4 + 4 reaches 64.10 ± 3.49% and chunk4 reaches 62.76 ± 4.42%.Source: §J.10 · p. 50

J.11 Addition feedback scale at a fixed learning-rate schedule

  • Reducing RLT-1 feedback scale does not improve addition accuracy at every width, and only three of 63 paired comparisons exceed both sample SDs.At 32 digits, RLT-1 means range from 14.44% to 15.91%, versus 16.84 ± 1.45% for Transformer 8.Source: §J.11 · p. 55, Table 10 · p. 55, Figure 28 · p. 56

Related papers

Loading 2610.07591v1…